Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:22.400949 16872 test_util.cc:268] Using random seed: 1033851473
I20240502 21:25:22.415098 16872 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:22.415185 16872 ts_itest-base.cc:113] --------------
I20240502 21:25:22.415306 16872 ts_itest-base.cc:114] 6 tablet servers
I20240502 21:25:22.415364 16872 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:22.415442 16872 ts_itest-base.cc:116] --------------
2024-05-02T21:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:22Z Disabled control of system clock
I20240502 21:25:22.444007 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:41017
--webserver_interface=127.16.122.62
--webserver_port=0
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.62:41017
--location_mapping_cmd=/tmp/dist-test-task3Dzs1I/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:25:22.819293 16886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.819622 16886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.819744 16886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:22.827517 16886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:22.827663 16886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:22.827724 16886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.827802 16886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:22.827860 16886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:22.837790 16886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3Dzs1I/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.16.122.62:41017
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.122.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.839267 16886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.841086 16886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.853693 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.854140 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.855366 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.856266 16886 server_base.cc:1034] running on GCE node
I20240502 21:25:22.857358 16886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.859495 16886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.860733 16886 hybrid_clock.cc:648] HybridClock initialized: now 1714685122860687 us; error 50 us; skew 500 ppm
I20240502 21:25:22.861122 16886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.863624 16886 webserver.cc:460] Webserver started at http://127.16.122.62:46633/ using document root <none> and password file <none>
I20240502 21:25:22.864456 16886 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.864593 16886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.864936 16886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.867872 16886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "072464139bd54d21bce68f004b7f598d"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-p9xn"
I20240502 21:25:22.868685 16886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "072464139bd54d21bce68f004b7f598d"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-p9xn"
I20240502 21:25:22.874212 16886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:25:22.877723 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.879166 16886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:22.879338 16886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
uuid: "072464139bd54d21bce68f004b7f598d"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-p9xn"
I20240502 21:25:22.879545 16886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.902398 16886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:22.903340 16886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.903633 16886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.922724 16886 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.62:41017
I20240502 21:25:22.922732 16953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.62:41017 every 8 connection(s)
I20240502 21:25:22.924237 16886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:22.928242 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:22.934880 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 16886
I20240502 21:25:22.935204 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:22.939628 16954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d: Bootstrap starting.
I20240502 21:25:22.943650 16954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:22.945057 16954 log.cc:824] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:22.948459 16954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d: No bootstrap required, opened a new log
I20240502 21:25:22.957746 16954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072464139bd54d21bce68f004b7f598d" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 41017 } }
I20240502 21:25:22.958257 16954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:22.958423 16954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072464139bd54d21bce68f004b7f598d, State: Initialized, Role: FOLLOWER
I20240502 21:25:22.959192 16954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072464139bd54d21bce68f004b7f598d" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 41017 } }
I20240502 21:25:22.959483 16954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:22.959614 16954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:22.959792 16954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:22.962507 16954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072464139bd54d21bce68f004b7f598d" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 41017 } }
I20240502 21:25:22.963178 16954 leader_election.cc:302] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 072464139bd54d21bce68f004b7f598d; no voters:
I20240502 21:25:22.964000 16954 leader_election.cc:288] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:22.964251 16959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:22.965664 16959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [term 1 LEADER]: Becoming Leader. State: Replica: 072464139bd54d21bce68f004b7f598d, State: Running, Role: LEADER
I20240502 21:25:22.966221 16959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072464139bd54d21bce68f004b7f598d" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 41017 } }
I20240502 21:25:22.966567 16954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:22.971856 16960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "072464139bd54d21bce68f004b7f598d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072464139bd54d21bce68f004b7f598d" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 41017 } } }
I20240502 21:25:22.972261 16960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:22.975006 16961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 072464139bd54d21bce68f004b7f598d. Latest consensus state: current_term: 1 leader_uuid: "072464139bd54d21bce68f004b7f598d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072464139bd54d21bce68f004b7f598d" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 41017 } } }
I20240502 21:25:22.975466 16961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:22.985833 16965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:22.991521 16965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:23.001682 16965 catalog_manager.cc:1347] Generated new cluster ID: 2b3e4a6bf00a4bc0916f8760182f5a70
I20240502 21:25:23.001822 16965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:23.024817 16965 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:23.026661 16965 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:23.053023 16965 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 072464139bd54d21bce68f004b7f598d: Generated new TSK 0
I20240502 21:25:23.053972 16965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:23.073803 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.1:0
--local_ip_for_outbound_sockets=127.16.122.1
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:23.429919 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.430455 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.430675 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.439190 16979 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:23.439386 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.439563 16979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.1
I20240502 21:25:23.449774 16979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.1
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.451283 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.453155 16979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.466408 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.467339 16985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.468464 16987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.469552 16979 server_base.cc:1034] running on GCE node
I20240502 21:25:23.470431 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.471429 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.472630 16979 hybrid_clock.cc:648] HybridClock initialized: now 1714685123472577 us; error 44 us; skew 500 ppm
I20240502 21:25:23.472955 16979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.475306 16979 webserver.cc:460] Webserver started at http://127.16.122.1:43547/ using document root <none> and password file <none>
I20240502 21:25:23.476083 16979 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.476207 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.476559 16979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.479405 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5a2cdeb7351f4fffb44146f4d9365403"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-p9xn"
I20240502 21:25:23.480201 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5a2cdeb7351f4fffb44146f4d9365403"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-p9xn"
I20240502 21:25:23.485965 16979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 21:25:23.489571 16993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.491235 16979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:23.491439 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5a2cdeb7351f4fffb44146f4d9365403"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-p9xn"
I20240502 21:25:23.491688 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:23.530200 16979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:23.531183 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:23.531457 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:23.532454 16979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:23.534058 16979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:23.534147 16979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.534230 16979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:23.534286 16979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.570096 16979 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.1:40947
I20240502 21:25:23.570168 17107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.1:40947 every 8 connection(s)
I20240502 21:25:23.571496 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:23.574196 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 16979
I20240502 21:25:23.574426 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:23.578262 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.2:0
--local_ip_for_outbound_sockets=127.16.122.2
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:23.586655 17108 heartbeater.cc:347] Connected to a master server at 127.16.122.62:41017
I20240502 21:25:23.587059 17108 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:23.588158 17108 heartbeater.cc:511] Master 127.16.122.62:41017 requested a full tablet report, sending...
I20240502 21:25:23.639905 16918 ts_manager.cc:194] Registered new tserver with Master: 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:23.641883 16918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.1:38697
W20240502 21:25:23.929344 17112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.929752 17112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.929893 17112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.942257 17112 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:23.942456 17112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.942665 17112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.2
I20240502 21:25:23.953855 17112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.2
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.955276 17112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.957041 17112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.970062 17119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.971112 17121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.972286 17118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.973510 17112 server_base.cc:1034] running on GCE node
I20240502 21:25:23.974150 17112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.975250 17112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.976485 17112 hybrid_clock.cc:648] HybridClock initialized: now 1714685123976446 us; error 31 us; skew 500 ppm
I20240502 21:25:23.976847 17112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.984596 17112 webserver.cc:460] Webserver started at http://127.16.122.2:42429/ using document root <none> and password file <none>
I20240502 21:25:23.985550 17112 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.985671 17112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.986009 17112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.988885 17112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-p9xn"
I20240502 21:25:23.989698 17112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-p9xn"
I20240502 21:25:23.996093 17112 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 21:25:24.003079 17127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.005211 17112 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20240502 21:25:24.005435 17112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-p9xn"
I20240502 21:25:24.005739 17112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:24.041030 17112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:24.041963 17112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:24.042265 17112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:24.043365 17112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:24.045059 17112 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:24.045174 17112 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.045293 17112 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:24.045369 17112 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.079739 17112 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.2:45287
I20240502 21:25:24.079811 17244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.2:45287 every 8 connection(s)
I20240502 21:25:24.081158 17112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:24.090157 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 17112
I20240502 21:25:24.090392 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:24.094494 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.3:0
--local_ip_for_outbound_sockets=127.16.122.3
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:24.095459 17245 heartbeater.cc:347] Connected to a master server at 127.16.122.62:41017
I20240502 21:25:24.095868 17245 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:24.097056 17245 heartbeater.cc:511] Master 127.16.122.62:41017 requested a full tablet report, sending...
I20240502 21:25:24.141721 16918 ts_manager.cc:194] Registered new tserver with Master: ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:24.142757 16918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.2:44797
W20240502 21:25:24.438758 17248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:24.439116 17248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:24.439277 17248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:24.447393 17248 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:24.447566 17248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:24.447753 17248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.3
I20240502 21:25:24.457532 17248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.3
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:24.459048 17248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:24.460870 17248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:24.473873 17254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.475069 17255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.476157 17257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:24.477267 17248 server_base.cc:1034] running on GCE node
I20240502 21:25:24.477978 17248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:24.479079 17248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:24.480314 17248 hybrid_clock.cc:648] HybridClock initialized: now 1714685124480267 us; error 45 us; skew 500 ppm
I20240502 21:25:24.480659 17248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:24.483956 17248 webserver.cc:460] Webserver started at http://127.16.122.3:33629/ using document root <none> and password file <none>
I20240502 21:25:24.484822 17248 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:24.484975 17248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:24.485342 17248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:24.488437 17248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "967c7c03fe754179ab1544965af0cf8d"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-p9xn"
I20240502 21:25:24.489223 17248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "967c7c03fe754179ab1544965af0cf8d"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-p9xn"
I20240502 21:25:24.494643 17248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240502 21:25:24.498221 17263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.499650 17248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:24.499789 17248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "967c7c03fe754179ab1544965af0cf8d"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-p9xn"
I20240502 21:25:24.499980 17248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:24.515743 17248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:24.516685 17248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:24.516973 17248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:24.518064 17248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:24.519773 17248 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:24.519866 17248 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.519984 17248 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:24.520054 17248 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.555611 17248 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.3:39535
I20240502 21:25:24.555673 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.3:39535 every 8 connection(s)
I20240502 21:25:24.557066 17248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:24.564986 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 17248
I20240502 21:25:24.565218 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:24.568984 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.4:0
--local_ip_for_outbound_sockets=127.16.122.4
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:24.571647 17380 heartbeater.cc:347] Connected to a master server at 127.16.122.62:41017
I20240502 21:25:24.572032 17380 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:24.573150 17380 heartbeater.cc:511] Master 127.16.122.62:41017 requested a full tablet report, sending...
I20240502 21:25:24.620105 16918 ts_manager.cc:194] Registered new tserver with Master: 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:24.621474 16918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.3:54485
I20240502 21:25:24.645270 17108 heartbeater.cc:503] Master 127.16.122.62:41017 was elected leader, sending a full tablet report...
W20240502 21:25:24.924685 17383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:24.925026 17383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:24.925168 17383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:24.932860 17383 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:24.932986 17383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:24.933168 17383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.4
I20240502 21:25:24.942571 17383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.4
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:24.944023 17383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:24.945843 17383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:24.959493 17389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.960690 17390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.961098 17392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:24.962478 17383 server_base.cc:1034] running on GCE node
I20240502 21:25:24.963165 17383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:24.964301 17383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:24.965567 17383 hybrid_clock.cc:648] HybridClock initialized: now 1714685124965513 us; error 39 us; skew 500 ppm
I20240502 21:25:24.965991 17383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:24.968936 17383 webserver.cc:460] Webserver started at http://127.16.122.4:41879/ using document root <none> and password file <none>
I20240502 21:25:24.969784 17383 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:24.969913 17383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:24.970248 17383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:24.973085 17383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8c66e3c3cad649daa465f3ae946f95f7"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-p9xn"
I20240502 21:25:24.973922 17383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8c66e3c3cad649daa465f3ae946f95f7"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-p9xn"
I20240502 21:25:24.979547 17383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 21:25:24.983325 17398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.984833 17383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:24.985002 17383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8c66e3c3cad649daa465f3ae946f95f7"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-p9xn"
I20240502 21:25:24.985224 17383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:25.012988 17383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:25.013986 17383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:25.014294 17383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:25.015444 17383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:25.017050 17383 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:25.017161 17383 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.017282 17383 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:25.017362 17383 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.050783 17383 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.4:36081
I20240502 21:25:25.050931 17513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.4:36081 every 8 connection(s)
I20240502 21:25:25.052223 17383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:25.061338 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 17383
I20240502 21:25:25.061544 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:25.065114 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.5:0
--local_ip_for_outbound_sockets=127.16.122.5
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:25.066170 17514 heartbeater.cc:347] Connected to a master server at 127.16.122.62:41017
I20240502 21:25:25.066509 17514 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:25.067409 17514 heartbeater.cc:511] Master 127.16.122.62:41017 requested a full tablet report, sending...
I20240502 21:25:25.116053 16918 ts_manager.cc:194] Registered new tserver with Master: 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:25.117167 16918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.4:51845
I20240502 21:25:25.145464 17245 heartbeater.cc:503] Master 127.16.122.62:41017 was elected leader, sending a full tablet report...
W20240502 21:25:25.415486 17517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:25.415819 17517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:25.415972 17517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:25.423758 17517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:25.423920 17517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:25.424095 17517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.5
I20240502 21:25:25.433501 17517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.5
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:25.434965 17517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:25.436764 17517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:25.449471 17523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.450431 17526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.451565 17524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:25.452606 17517 server_base.cc:1034] running on GCE node
I20240502 21:25:25.453336 17517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:25.454433 17517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:25.455657 17517 hybrid_clock.cc:648] HybridClock initialized: now 1714685125455607 us; error 47 us; skew 500 ppm
I20240502 21:25:25.456007 17517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:25.458434 17517 webserver.cc:460] Webserver started at http://127.16.122.5:41637/ using document root <none> and password file <none>
I20240502 21:25:25.459208 17517 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:25.459322 17517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:25.459647 17517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:25.462332 17517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "86a457a4beb4478b8a34e3633f8d2238"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-p9xn"
I20240502 21:25:25.463174 17517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "86a457a4beb4478b8a34e3633f8d2238"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-p9xn"
I20240502 21:25:25.468726 17517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:25.472430 17532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.473881 17517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:25.474064 17517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "86a457a4beb4478b8a34e3633f8d2238"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-p9xn"
I20240502 21:25:25.474316 17517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:25.505915 17517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:25.506963 17517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:25.507266 17517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:25.508347 17517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:25.510022 17517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:25.510124 17517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.510282 17517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:25.510390 17517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.544275 17517 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.5:36211
I20240502 21:25:25.544373 17647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.5:36211 every 8 connection(s)
I20240502 21:25:25.545655 17517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:25.546912 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 17517
I20240502 21:25:25.547163 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:25.551227 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.6:0
--local_ip_for_outbound_sockets=127.16.122.6
--webserver_interface=127.16.122.6
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--builtin_ntp_servers=127.16.122.20:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:25.562252 17648 heartbeater.cc:347] Connected to a master server at 127.16.122.62:41017
I20240502 21:25:25.562631 17648 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:25.563516 17648 heartbeater.cc:511] Master 127.16.122.62:41017 requested a full tablet report, sending...
I20240502 21:25:25.609954 16918 ts_manager.cc:194] Registered new tserver with Master: 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:25.611085 16918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.5:53803
I20240502 21:25:25.624351 17380 heartbeater.cc:503] Master 127.16.122.62:41017 was elected leader, sending a full tablet report...
W20240502 21:25:25.904542 17651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:25.904904 17651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:25.905030 17651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:25.912946 17651 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:25.913077 17651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:25.913197 17651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.6
I20240502 21:25:25.922995 17651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.122.6
--webserver_port=0
--tserver_master_addrs=127.16.122.62:41017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.6
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:25.924396 17651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:25.926193 17651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:25.938977 17658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.940021 17657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:25.941144 17660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:25.941982 17651 server_base.cc:1034] running on GCE node
I20240502 21:25:25.942705 17651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:25.943800 17651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:25.945044 17651 hybrid_clock.cc:648] HybridClock initialized: now 1714685125944980 us; error 61 us; skew 500 ppm
I20240502 21:25:25.945420 17651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:25.948012 17651 webserver.cc:460] Webserver started at http://127.16.122.6:40645/ using document root <none> and password file <none>
I20240502 21:25:25.948835 17651 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:25.948961 17651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:25.949290 17651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:25.952301 17651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "7181173c030d4a6db6e29c158fdb9a69"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-p9xn"
I20240502 21:25:25.953163 17651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "7181173c030d4a6db6e29c158fdb9a69"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-p9xn"
I20240502 21:25:25.959326 17651 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240502 21:25:25.963018 17666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.964509 17651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:25.964677 17651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7181173c030d4a6db6e29c158fdb9a69"
format_stamp: "Formatted at 2024-05-02 21:25:25 on dist-test-slave-p9xn"
I20240502 21:25:25.964895 17651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:25.991056 17651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:25.992003 17651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:25.992298 17651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:25.993388 17651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:25.995440 17651 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:25.995543 17651 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:25.995663 17651 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:25.995739 17651 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:26.032738 17651 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.6:35119
I20240502 21:25:26.032785 17780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.6:35119 every 8 connection(s)
I20240502 21:25:26.034193 17651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:26.043946 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 17651
I20240502 21:25:26.044253 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:25:26.050518 17781 heartbeater.cc:347] Connected to a master server at 127.16.122.62:41017
I20240502 21:25:26.050897 17781 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:26.051800 17781 heartbeater.cc:511] Master 127.16.122.62:41017 requested a full tablet report, sending...
I20240502 21:25:26.099634 16918 ts_manager.cc:194] Registered new tserver with Master: 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:26.100601 16918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.6:42465
I20240502 21:25:26.103914 16872 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 21:25:26.120433 17514 heartbeater.cc:503] Master 127.16.122.62:41017 was elected leader, sending a full tablet report...
I20240502 21:25:26.125919 16872 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 21:25:26.613952 17648 heartbeater.cc:503] Master 127.16.122.62:41017 was elected leader, sending a full tablet report...
I20240502 21:25:27.104199 17781 heartbeater.cc:503] Master 127.16.122.62:41017 was elected leader, sending a full tablet report...
I20240502 21:25:29.879895 16918 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:29.934051 17176 tablet_service.cc:1467] Processing CreateTablet for tablet c8924e38adc84a4fb43d7c1daf41dce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.937297 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8924e38adc84a4fb43d7c1daf41dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.938249 17716 tablet_service.cc:1467] Processing CreateTablet for tablet c8924e38adc84a4fb43d7c1daf41dce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.938249 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 66416d27cc074e2a899d80a6bb444f87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.939738 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66416d27cc074e2a899d80a6bb444f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.939738 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8924e38adc84a4fb43d7c1daf41dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.930213 17448 tablet_service.cc:1467] Processing CreateTablet for tablet c8924e38adc84a4fb43d7c1daf41dce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.942811 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8924e38adc84a4fb43d7c1daf41dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.943272 17580 tablet_service.cc:1467] Processing CreateTablet for tablet f0993ddf93d545cf8de9d26bd2085556 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:29.943778 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0993ddf93d545cf8de9d26bd2085556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:29.981433 17807 tablet_bootstrap.cc:492] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:29.988536 17807 tablet_bootstrap.cc:654] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.993189 17807 log.cc:824] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:29.995374 17805 tablet_bootstrap.cc:492] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:30.000010 17807 tablet_bootstrap.cc:492] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:29.997222 17806 tablet_bootstrap.cc:492] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:30.000404 17807 ts_tablet_manager.cc:1367] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20240502 21:25:30.005069 17805 tablet_bootstrap.cc:654] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:29.992924 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 66416d27cc074e2a899d80a6bb444f87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.007212 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66416d27cc074e2a899d80a6bb444f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.008250 17805 log.cc:824] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:30.008682 17313 tablet_service.cc:1467] Processing CreateTablet for tablet f0993ddf93d545cf8de9d26bd2085556 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.009194 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0993ddf93d545cf8de9d26bd2085556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.009498 17806 tablet_bootstrap.cc:654] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.010094 17807 raft_consensus.cc:348] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.010684 17807 raft_consensus.cc:374] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.010846 17807 raft_consensus.cc:729] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.011735 17807 consensus_queue.cc:260] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.012090 17806 log.cc:824] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:30.019495 17806 tablet_bootstrap.cc:492] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:30.022244 17806 ts_tablet_manager.cc:1367] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.007s
I20240502 21:25:30.024813 17805 tablet_bootstrap.cc:492] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:30.025279 17805 ts_tablet_manager.cc:1367] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20240502 21:25:30.031234 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 66416d27cc074e2a899d80a6bb444f87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.033269 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66416d27cc074e2a899d80a6bb444f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.038465 17042 tablet_service.cc:1467] Processing CreateTablet for tablet f0993ddf93d545cf8de9d26bd2085556 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f728b788656b40dd8a9cf93ce9cdf0d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:30.039191 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0993ddf93d545cf8de9d26bd2085556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.040167 17807 ts_tablet_manager.cc:1398] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.040s user 0.025s sys 0.012s
I20240502 21:25:30.045110 17806 raft_consensus.cc:348] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.049993 17805 raft_consensus.cc:348] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.050736 17806 raft_consensus.cc:374] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.050905 17806 raft_consensus.cc:729] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.051729 17806 consensus_queue.cc:260] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.052121 17805 raft_consensus.cc:374] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.052569 17805 raft_consensus.cc:729] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.053391 17805 consensus_queue.cc:260] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
W20240502 21:25:30.057762 17515 tablet.cc:2369] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:30.061991 17805 ts_tablet_manager.cc:1398] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.037s user 0.018s sys 0.000s
I20240502 21:25:30.065165 17806 ts_tablet_manager.cc:1398] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.033s user 0.008s sys 0.006s
I20240502 21:25:30.065866 17806 tablet_bootstrap.cc:492] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:30.068971 17806 tablet_bootstrap.cc:654] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.071534 17814 tablet_bootstrap.cc:492] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:30.071830 17813 raft_consensus.cc:482] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.072119 17813 raft_consensus.cc:504] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.072432 17809 tablet_bootstrap.cc:492] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:30.076581 17809 tablet_bootstrap.cc:654] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.076761 17814 tablet_bootstrap.cc:654] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.077500 17816 tablet_bootstrap.cc:492] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:30.078740 17809 log.cc:824] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:30.082969 17806 tablet_bootstrap.cc:492] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:30.083145 17806 ts_tablet_manager.cc:1367] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.002s
I20240502 21:25:30.084205 17806 raft_consensus.cc:348] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.084424 17806 raft_consensus.cc:374] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.084513 17806 raft_consensus.cc:729] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.084739 17806 consensus_queue.cc:260] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.078333 17814 log.cc:824] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403: Log is configured to *not* fsync() on all Append() calls
W20240502 21:25:30.087844 17246 tablet.cc:2369] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:30.091522 17816 tablet_bootstrap.cc:654] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.094898 17813 leader_election.cc:288] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:30.105127 17806 ts_tablet_manager.cc:1398] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.022s user 0.008s sys 0.004s
I20240502 21:25:30.106600 17816 log.cc:824] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:30.109862 17809 tablet_bootstrap.cc:492] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:30.109862 17814 tablet_bootstrap.cc:492] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:30.110213 17809 ts_tablet_manager.cc:1367] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.027s
I20240502 21:25:30.110234 17814 ts_tablet_manager.cc:1367] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.013s
I20240502 21:25:30.105072 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66416d27cc074e2a899d80a6bb444f87" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
W20240502 21:25:30.113410 17534 leader_election.cc:341] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:30.114933 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66416d27cc074e2a899d80a6bb444f87" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
W20240502 21:25:30.116719 17534 leader_election.cc:341] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:30.116897 17534 leader_election.cc:302] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238; no voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d
I20240502 21:25:30.117772 17820 raft_consensus.cc:2726] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:30.118289 17816 tablet_bootstrap.cc:492] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:30.118599 17816 ts_tablet_manager.cc:1367] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.003s
I20240502 21:25:30.120613 17814 raft_consensus.cc:348] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.121026 17814 raft_consensus.cc:374] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.121145 17814 raft_consensus.cc:729] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.121861 17814 consensus_queue.cc:260] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.121943 17809 raft_consensus.cc:348] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.122411 17809 raft_consensus.cc:374] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.122543 17809 raft_consensus.cc:729] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.122573 17811 raft_consensus.cc:482] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.122820 17811 raft_consensus.cc:504] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.123376 17809 consensus_queue.cc:260] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.124055 17816 raft_consensus.cc:348] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.124413 17816 raft_consensus.cc:374] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.124519 17816 raft_consensus.cc:729] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.125001 17811 leader_election.cc:288] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:30.125275 17816 consensus_queue.cc:260] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.134238 17816 ts_tablet_manager.cc:1398] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.015s user 0.004s sys 0.009s
I20240502 21:25:30.140074 17816 tablet_bootstrap.cc:492] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:30.142422 17816 tablet_bootstrap.cc:654] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.151901 17816 tablet_bootstrap.cc:492] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:30.152096 17816 ts_tablet_manager.cc:1367] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20240502 21:25:30.153177 17816 raft_consensus.cc:348] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.153465 17816 raft_consensus.cc:374] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.153575 17816 raft_consensus.cc:729] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.153908 17816 consensus_queue.cc:260] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.157369 17816 ts_tablet_manager.cc:1398] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:25:30.158982 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8924e38adc84a4fb43d7c1daf41dce3" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:30.159703 17197 raft_consensus.cc:2454] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:30.160873 17401 leader_election.cc:302] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:30.162128 17809 ts_tablet_manager.cc:1398] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.052s user 0.012s sys 0.036s
I20240502 21:25:30.162398 17811 raft_consensus.cc:2781] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.162474 17814 ts_tablet_manager.cc:1398] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.052s user 0.024s sys 0.025s
I20240502 21:25:30.162669 17811 raft_consensus.cc:482] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.162815 17811 raft_consensus.cc:3037] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.162951 17814 tablet_bootstrap.cc:492] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:30.164039 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8924e38adc84a4fb43d7c1daf41dce3" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:30.164731 17736 raft_consensus.cc:2454] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:30.166061 17811 raft_consensus.cc:504] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.166644 17814 tablet_bootstrap.cc:654] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.166836 17811 leader_election.cc:288] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:30.168037 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8924e38adc84a4fb43d7c1daf41dce3" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:30.168304 17197 raft_consensus.cc:3037] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.168491 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8924e38adc84a4fb43d7c1daf41dce3" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:30.168687 17736 raft_consensus.cc:3037] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.170030 17814 tablet_bootstrap.cc:492] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:30.170203 17814 ts_tablet_manager.cc:1367] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 21:25:30.171263 17197 raft_consensus.cc:2454] T c8924e38adc84a4fb43d7c1daf41dce3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:30.171480 17814 raft_consensus.cc:348] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.171710 17814 raft_consensus.cc:374] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.171715 17401 leader_election.cc:302] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:30.171816 17814 raft_consensus.cc:729] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.172053 17811 raft_consensus.cc:2781] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.172091 17814 consensus_queue.cc:260] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.172539 17736 raft_consensus.cc:2454] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:30.173403 17811 raft_consensus.cc:686] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:30.173496 17814 ts_tablet_manager.cc:1398] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:25:30.174026 17811 consensus_queue.cc:237] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:30.180389 16918 catalog_manager.cc:5484] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:30.289261 17783 tablet.cc:2369] T c8924e38adc84a4fb43d7c1daf41dce3 P 7181173c030d4a6db6e29c158fdb9a69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:30.301365 17649 tablet.cc:2369] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:30.312927 17381 tablet.cc:2369] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:30.328429 17109 tablet.cc:2369] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:30.395903 17820 raft_consensus.cc:482] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.396215 17820 raft_consensus.cc:504] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.397090 17820 leader_election.cc:288] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:30.397760 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0993ddf93d545cf8de9d26bd2085556" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:30.397845 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0993ddf93d545cf8de9d26bd2085556" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:30.398162 17062 raft_consensus.cc:2454] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:30.398206 17334 raft_consensus.cc:2454] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:30.398653 17534 leader_election.cc:302] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:30.398950 17820 raft_consensus.cc:2781] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.399067 17820 raft_consensus.cc:482] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.399179 17820 raft_consensus.cc:3037] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.401401 17820 raft_consensus.cc:504] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.401933 17820 leader_election.cc:288] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:30.402312 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0993ddf93d545cf8de9d26bd2085556" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:30.402458 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0993ddf93d545cf8de9d26bd2085556" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:30.402571 17062 raft_consensus.cc:3037] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.402673 17334 raft_consensus.cc:3037] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.404789 17062 raft_consensus.cc:2454] T f0993ddf93d545cf8de9d26bd2085556 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:30.405208 17534 leader_election.cc:302] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:30.405457 17334 raft_consensus.cc:2454] T f0993ddf93d545cf8de9d26bd2085556 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:30.405576 17820 raft_consensus.cc:2781] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.405805 17820 raft_consensus.cc:686] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:30.406370 17820 consensus_queue.cc:237] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:30.411481 16918 catalog_manager.cc:5484] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.444352 17826 raft_consensus.cc:482] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:30.444547 17826 raft_consensus.cc:504] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.446215 17826 leader_election.cc:288] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:30.457516 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66416d27cc074e2a899d80a6bb444f87" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:30.457906 17062 raft_consensus.cc:2454] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:30.457932 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66416d27cc074e2a899d80a6bb444f87" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:30.458387 17602 raft_consensus.cc:2454] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:30.458624 17265 leader_election.cc:302] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:30.459153 17826 raft_consensus.cc:2781] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:30.459300 17826 raft_consensus.cc:482] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:30.459429 17826 raft_consensus.cc:3037] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.462098 17826 raft_consensus.cc:504] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.462951 17826 leader_election.cc:288] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Requested vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:30.463481 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66416d27cc074e2a899d80a6bb444f87" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:30.463544 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66416d27cc074e2a899d80a6bb444f87" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:30.463773 17062 raft_consensus.cc:3037] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.463785 17602 raft_consensus.cc:3037] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.467029 17602 raft_consensus.cc:2454] T 66416d27cc074e2a899d80a6bb444f87 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:30.467370 17062 raft_consensus.cc:2454] T 66416d27cc074e2a899d80a6bb444f87 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:30.467510 17266 leader_election.cc:302] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:30.467908 17826 raft_consensus.cc:2781] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.468556 17826 raft_consensus.cc:686] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [term 1 LEADER]: Becoming Leader. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Running, Role: LEADER
I20240502 21:25:30.469276 17826 consensus_queue.cc:237] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:30.474407 16918 catalog_manager.cc:5484] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 967c7c03fe754179ab1544965af0cf8d (127.16.122.3). New cstate: current_term: 1 leader_uuid: "967c7c03fe754179ab1544965af0cf8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:30.693163 17840 consensus_queue.cc:1035] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.701675 17843 consensus_queue.cc:1035] T c8924e38adc84a4fb43d7c1daf41dce3 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.850710 17820 consensus_queue.cc:1035] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.859203 17820 consensus_queue.cc:1035] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:30.943179 17839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:30.943553 17839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:30.951119 17839 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:30.966939 17826 consensus_queue.cc:1035] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:30.972985 17826 consensus_queue.cc:1035] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 309.575 ms
time per row: 0.154787 ms
I20240502 21:25:31.418802 16915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:31.428231 17176 tablet_service.cc:1467] Processing CreateTablet for tablet a27298e93e18457ca4cc29dc87b764c9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.428794 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27298e93e18457ca4cc29dc87b764c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.428941 17447 tablet_service.cc:1467] Processing CreateTablet for tablet bb83ef0f3aa84014afb1eb0e664e8ef3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.429009 17448 tablet_service.cc:1467] Processing CreateTablet for tablet a27298e93e18457ca4cc29dc87b764c9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.428807 17175 tablet_service.cc:1467] Processing CreateTablet for tablet bb83ef0f3aa84014afb1eb0e664e8ef3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.429507 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb83ef0f3aa84014afb1eb0e664e8ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.429656 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb83ef0f3aa84014afb1eb0e664e8ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.429673 17716 tablet_service.cc:1467] Processing CreateTablet for tablet a27298e93e18457ca4cc29dc87b764c9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.430119 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27298e93e18457ca4cc29dc87b764c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.435226 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 6e770c72315d452292acca3919362b00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.435789 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e770c72315d452292acca3919362b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.436865 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27298e93e18457ca4cc29dc87b764c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.443545 17915 tablet_bootstrap.cc:492] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:31.443884 17715 tablet_service.cc:1467] Processing CreateTablet for tablet bb83ef0f3aa84014afb1eb0e664e8ef3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.444506 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb83ef0f3aa84014afb1eb0e664e8ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.445168 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 6e770c72315d452292acca3919362b00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.445613 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e770c72315d452292acca3919362b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.447857 17915 tablet_bootstrap.cc:654] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.450984 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 6e770c72315d452292acca3919362b00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9b93dba10aa749299ee73472c4537b42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:31.451539 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e770c72315d452292acca3919362b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:31.455562 17914 tablet_bootstrap.cc:492] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:31.455605 17918 tablet_bootstrap.cc:492] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:31.456707 17915 tablet_bootstrap.cc:492] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:31.456918 17915 ts_tablet_manager.cc:1367] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240502 21:25:31.457530 17914 tablet_bootstrap.cc:654] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.458025 17916 tablet_bootstrap.cc:492] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:31.458096 17915 raft_consensus.cc:348] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.458673 17915 raft_consensus.cc:374] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.458818 17915 raft_consensus.cc:729] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.459152 17915 consensus_queue.cc:260] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.460019 17915 ts_tablet_manager.cc:1398] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:31.460114 17916 tablet_bootstrap.cc:654] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.460811 17915 tablet_bootstrap.cc:492] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:31.461894 17918 tablet_bootstrap.cc:654] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.462847 17915 tablet_bootstrap.cc:654] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.465680 17843 raft_consensus.cc:482] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.465885 17916 tablet_bootstrap.cc:492] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:31.466066 17916 ts_tablet_manager.cc:1367] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:25:31.465902 17843 raft_consensus.cc:504] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.466933 17843 leader_election.cc:288] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:31.467469 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb83ef0f3aa84014afb1eb0e664e8ef3" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:31.467654 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb83ef0f3aa84014afb1eb0e664e8ef3" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:31.467851 17915 tablet_bootstrap.cc:492] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:31.468011 17915 ts_tablet_manager.cc:1367] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:25:31.468501 17916 raft_consensus.cc:348] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
W20240502 21:25:31.468601 17401 leader_election.cc:341] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:31.468756 17916 raft_consensus.cc:374] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.468856 17916 raft_consensus.cc:729] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
W20240502 21:25:31.468896 17401 leader_election.cc:341] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:31.468967 17915 raft_consensus.cc:348] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.469028 17401 leader_election.cc:302] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7; no voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820
I20240502 21:25:31.469208 17915 raft_consensus.cc:374] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.469293 17915 raft_consensus.cc:729] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.469157 17916 consensus_queue.cc:260] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.469360 17843 raft_consensus.cc:2726] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:31.469542 17915 consensus_queue.cc:260] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.470304 17915 ts_tablet_manager.cc:1398] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:31.471529 17914 tablet_bootstrap.cc:492] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:31.471760 17914 ts_tablet_manager.cc:1367] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20240502 21:25:31.473388 17914 raft_consensus.cc:348] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.473786 17914 raft_consensus.cc:374] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.473922 17914 raft_consensus.cc:729] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.474218 17914 consensus_queue.cc:260] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.474746 17918 tablet_bootstrap.cc:492] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:31.474985 17918 ts_tablet_manager.cc:1367] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240502 21:25:31.478200 17918 raft_consensus.cc:348] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.478665 17918 raft_consensus.cc:374] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.478848 17918 raft_consensus.cc:729] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.479250 17918 consensus_queue.cc:260] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.480505 17918 ts_tablet_manager.cc:1398] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:31.481266 17916 ts_tablet_manager.cc:1398] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20240502 21:25:31.481745 17916 tablet_bootstrap.cc:492] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:31.486295 17916 tablet_bootstrap.cc:654] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.490280 17926 tablet_bootstrap.cc:492] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:31.490955 17914 ts_tablet_manager.cc:1398] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20240502 21:25:31.491407 17914 tablet_bootstrap.cc:492] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:31.495095 17914 tablet_bootstrap.cc:654] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.497020 17924 tablet_bootstrap.cc:492] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:31.498618 17916 tablet_bootstrap.cc:492] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:31.498618 17914 tablet_bootstrap.cc:492] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:31.498788 17914 ts_tablet_manager.cc:1367] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:25:31.498800 17916 ts_tablet_manager.cc:1367] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20240502 21:25:31.499162 17926 tablet_bootstrap.cc:654] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.499174 17924 tablet_bootstrap.cc:654] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:31.499564 17916 raft_consensus.cc:348] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.499560 17914 raft_consensus.cc:348] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.499789 17916 raft_consensus.cc:374] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.499790 17914 raft_consensus.cc:374] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.499862 17914 raft_consensus.cc:729] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.499862 17916 raft_consensus.cc:729] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.500171 17914 consensus_queue.cc:260] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.500142 17916 consensus_queue.cc:260] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.500730 17916 ts_tablet_manager.cc:1398] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:31.500828 17914 ts_tablet_manager.cc:1398] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:31.502802 17926 tablet_bootstrap.cc:492] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:31.503171 17926 ts_tablet_manager.cc:1367] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20240502 21:25:31.503249 17924 tablet_bootstrap.cc:492] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:31.503480 17924 ts_tablet_manager.cc:1367] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 21:25:31.504397 17926 raft_consensus.cc:348] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.504473 17924 raft_consensus.cc:348] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.504657 17926 raft_consensus.cc:374] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.504763 17926 raft_consensus.cc:729] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.504825 17924 raft_consensus.cc:374] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:31.505146 17924 raft_consensus.cc:729] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:31.505113 17926 consensus_queue.cc:260] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.505527 17924 consensus_queue.cc:260] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.506064 17924 ts_tablet_manager.cc:1398] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:31.506072 17926 ts_tablet_manager.cc:1398] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:31.509552 17925 raft_consensus.cc:482] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.509773 17925 raft_consensus.cc:504] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.511755 17925 leader_election.cc:288] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:31.518898 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb83ef0f3aa84014afb1eb0e664e8ef3" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:31.519332 17468 raft_consensus.cc:2454] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:31.519948 17131 leader_election.cc:302] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:31.520470 17925 raft_consensus.cc:2781] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:31.520612 17925 raft_consensus.cc:482] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:31.520705 17925 raft_consensus.cc:3037] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.522241 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb83ef0f3aa84014afb1eb0e664e8ef3" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:31.522317 17820 raft_consensus.cc:482] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.522567 17735 raft_consensus.cc:2454] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:31.522544 17820 raft_consensus.cc:504] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.523231 17925 raft_consensus.cc:504] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.523392 17820 leader_election.cc:288] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:31.523908 17925 leader_election.cc:288] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:31.523932 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770c72315d452292acca3919362b00" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:31.524000 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770c72315d452292acca3919362b00" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:31.524204 17062 raft_consensus.cc:2454] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:31.524262 17334 raft_consensus.cc:2454] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:31.524344 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb83ef0f3aa84014afb1eb0e664e8ef3" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:31.524545 17468 raft_consensus.cc:3037] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.524781 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb83ef0f3aa84014afb1eb0e664e8ef3" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:31.524861 17534 leader_election.cc:302] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:31.524987 17735 raft_consensus.cc:3037] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.525147 17820 raft_consensus.cc:2781] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:31.525283 17820 raft_consensus.cc:482] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:31.525379 17820 raft_consensus.cc:3037] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.527280 17820 raft_consensus.cc:504] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.527530 17468 raft_consensus.cc:2454] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:31.527856 17735 raft_consensus.cc:2454] T bb83ef0f3aa84014afb1eb0e664e8ef3 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:31.527869 17820 leader_election.cc:288] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:31.527968 17131 leader_election.cc:302] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:31.528295 17925 raft_consensus.cc:2781] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:31.528575 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770c72315d452292acca3919362b00" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:31.528816 17062 raft_consensus.cc:3037] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.530669 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770c72315d452292acca3919362b00" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:31.530903 17062 raft_consensus.cc:2454] T 6e770c72315d452292acca3919362b00 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:31.530946 17334 raft_consensus.cc:3037] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.531309 17534 leader_election.cc:302] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:31.531565 17820 raft_consensus.cc:2781] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:31.531713 17820 raft_consensus.cc:686] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:31.532035 17820 consensus_queue.cc:237] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:31.534721 17334 raft_consensus.cc:2454] T 6e770c72315d452292acca3919362b00 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:31.534819 16914 catalog_manager.cc:5484] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.535285 17925 raft_consensus.cc:686] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:31.535799 17925 consensus_queue.cc:237] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.539950 16914 catalog_manager.cc:5484] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.749821 17925 raft_consensus.cc:482] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:31.750123 17925 raft_consensus.cc:504] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.750870 17925 leader_election.cc:288] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:31.751423 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27298e93e18457ca4cc29dc87b764c9" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:31.751461 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27298e93e18457ca4cc29dc87b764c9" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:31.751660 17468 raft_consensus.cc:2454] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:31.751681 17735 raft_consensus.cc:2454] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:31.752090 17129 leader_election.cc:302] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:31.752388 17925 raft_consensus.cc:2781] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:31.752471 17925 raft_consensus.cc:482] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:31.752547 17925 raft_consensus.cc:3037] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.754513 17925 raft_consensus.cc:504] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.755066 17925 leader_election.cc:288] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:31.755503 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27298e93e18457ca4cc29dc87b764c9" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:31.755686 17468 raft_consensus.cc:3037] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.755617 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27298e93e18457ca4cc29dc87b764c9" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:31.755841 17735 raft_consensus.cc:3037] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:31.757506 17468 raft_consensus.cc:2454] T a27298e93e18457ca4cc29dc87b764c9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:31.757701 17735 raft_consensus.cc:2454] T a27298e93e18457ca4cc29dc87b764c9 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:31.757944 17131 leader_election.cc:302] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:31.758296 17925 raft_consensus.cc:2781] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:31.758493 17925 raft_consensus.cc:686] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:31.758746 17925 consensus_queue.cc:237] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:31.761461 16914 catalog_manager.cc:5484] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:31.928253 17925 consensus_queue.cc:1035] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:31.935559 17925 consensus_queue.cc:1035] T bb83ef0f3aa84014afb1eb0e664e8ef3 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:32.019232 17820 consensus_queue.cc:1035] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:32.027233 17949 consensus_queue.cc:1035] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:32.196807 17939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.197217 17939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:25:32.202298 17935 consensus_queue.cc:1035] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:32.206913 17939 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:32.207937 17925 consensus_queue.cc:1035] T a27298e93e18457ca4cc29dc87b764c9 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 219.194 ms
time per row: 0.109597 ms
I20240502 21:25:32.589461 16915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:32.600637 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 7553a4d675be4edb9c42f655b8e439da (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.600775 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 7553a4d675be4edb9c42f655b8e439da (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.601167 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7553a4d675be4edb9c42f655b8e439da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.601289 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7553a4d675be4edb9c42f655b8e439da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.601749 17313 tablet_service.cc:1467] Processing CreateTablet for tablet cbfe08cd1c944e3f9a5438d204dfd196 (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.601838 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 7553a4d675be4edb9c42f655b8e439da (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.602144 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfe08cd1c944e3f9a5438d204dfd196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.602277 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7553a4d675be4edb9c42f655b8e439da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.606817 17176 tablet_service.cc:1467] Processing CreateTablet for tablet ce5736b1f933416fb88223e3557cf57e (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.607262 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce5736b1f933416fb88223e3557cf57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.607589 17448 tablet_service.cc:1467] Processing CreateTablet for tablet ce5736b1f933416fb88223e3557cf57e (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.608062 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce5736b1f933416fb88223e3557cf57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.610786 17580 tablet_service.cc:1467] Processing CreateTablet for tablet cbfe08cd1c944e3f9a5438d204dfd196 (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.611168 17716 tablet_service.cc:1467] Processing CreateTablet for tablet ce5736b1f933416fb88223e3557cf57e (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.611335 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfe08cd1c944e3f9a5438d204dfd196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.611744 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce5736b1f933416fb88223e3557cf57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.615051 17041 tablet_service.cc:1467] Processing CreateTablet for tablet cbfe08cd1c944e3f9a5438d204dfd196 (DEFAULT_TABLE table=rebalance_test_table_2 [id=77cef58c189647fd86f99e6f13527a53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:32.615530 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfe08cd1c944e3f9a5438d204dfd196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.629734 18006 tablet_bootstrap.cc:492] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:32.630226 18007 tablet_bootstrap.cc:492] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:32.632793 18007 tablet_bootstrap.cc:654] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.632890 18003 tablet_bootstrap.cc:492] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:32.634316 18008 tablet_bootstrap.cc:492] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:32.634971 18005 tablet_bootstrap.cc:492] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:32.635238 18006 tablet_bootstrap.cc:654] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.636631 18008 tablet_bootstrap.cc:654] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.637197 18005 tablet_bootstrap.cc:654] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.638303 18004 tablet_bootstrap.cc:492] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:32.639513 18003 tablet_bootstrap.cc:654] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.641120 18004 tablet_bootstrap.cc:654] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.641906 18008 tablet_bootstrap.cc:492] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:32.642091 18008 ts_tablet_manager.cc:1367] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 21:25:32.642125 18007 tablet_bootstrap.cc:492] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:32.642305 18007 ts_tablet_manager.cc:1367] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 21:25:32.643222 18008 raft_consensus.cc:348] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.643432 18007 raft_consensus.cc:348] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.643643 18008 raft_consensus.cc:374] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.643793 18008 raft_consensus.cc:729] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.643819 18007 raft_consensus.cc:374] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.643968 18007 raft_consensus.cc:729] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.644306 18007 consensus_queue.cc:260] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.644440 18008 consensus_queue.cc:260] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.645305 18007 ts_tablet_manager.cc:1398] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:32.649339 18004 tablet_bootstrap.cc:492] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:32.649585 18004 ts_tablet_manager.cc:1367] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 21:25:32.650501 18004 raft_consensus.cc:348] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.650684 18004 raft_consensus.cc:374] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.650813 18004 raft_consensus.cc:729] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.651048 18004 consensus_queue.cc:260] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.651742 18004 ts_tablet_manager.cc:1398] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:32.652010 18004 tablet_bootstrap.cc:492] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:32.652150 18005 tablet_bootstrap.cc:492] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:32.652354 18005 ts_tablet_manager.cc:1367] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240502 21:25:32.653524 18004 tablet_bootstrap.cc:654] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.653393 18005 raft_consensus.cc:348] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.653928 18005 raft_consensus.cc:374] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.654050 18005 raft_consensus.cc:729] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.654381 18005 consensus_queue.cc:260] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.655526 18005 ts_tablet_manager.cc:1398] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:32.655634 18004 tablet_bootstrap.cc:492] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:32.655800 18003 tablet_bootstrap.cc:492] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:32.655902 18004 ts_tablet_manager.cc:1367] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:32.655974 18003 ts_tablet_manager.cc:1367] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.006s
I20240502 21:25:32.656800 18004 raft_consensus.cc:348] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.656816 18003 raft_consensus.cc:348] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.656991 18006 tablet_bootstrap.cc:492] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:32.657009 18003 raft_consensus.cc:374] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.657095 18003 raft_consensus.cc:729] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.657179 18006 ts_tablet_manager.cc:1367] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20240502 21:25:32.657266 18004 raft_consensus.cc:374] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.657450 18004 raft_consensus.cc:729] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.657333 18003 consensus_queue.cc:260] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.657856 18004 consensus_queue.cc:260] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.658438 18003 ts_tablet_manager.cc:1398] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:25:32.658658 18004 ts_tablet_manager.cc:1398] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:32.658743 18003 tablet_bootstrap.cc:492] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:32.659325 18006 raft_consensus.cc:348] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.659574 18006 raft_consensus.cc:374] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.659688 18006 raft_consensus.cc:729] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.660023 18006 consensus_queue.cc:260] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.660240 18003 tablet_bootstrap.cc:654] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.660780 18006 ts_tablet_manager.cc:1398] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:32.661301 18008 ts_tablet_manager.cc:1398] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240502 21:25:32.666903 18005 tablet_bootstrap.cc:492] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:32.668654 18005 tablet_bootstrap.cc:654] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.670794 18005 tablet_bootstrap.cc:492] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:32.670974 18005 ts_tablet_manager.cc:1367] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:32.671056 18003 tablet_bootstrap.cc:492] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:32.671236 18003 ts_tablet_manager.cc:1367] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240502 21:25:32.671795 18005 raft_consensus.cc:348] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.671972 18005 raft_consensus.cc:374] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.671880 18003 raft_consensus.cc:348] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.672048 18005 raft_consensus.cc:729] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.672091 18003 raft_consensus.cc:374] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.672178 18003 raft_consensus.cc:729] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.672281 18005 consensus_queue.cc:260] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.672415 18003 consensus_queue.cc:260] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.672814 18005 ts_tablet_manager.cc:1398] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:32.673494 18003 ts_tablet_manager.cc:1398] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:32.772078 17867 raft_consensus.cc:482] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:32.772269 17867 raft_consensus.cc:504] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.772994 17867 leader_election.cc:288] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:32.773654 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7553a4d675be4edb9c42f655b8e439da" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:32.773677 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7553a4d675be4edb9c42f655b8e439da" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:32.773926 17062 raft_consensus.cc:2454] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:32.773950 17602 raft_consensus.cc:2454] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:32.774467 17265 leader_election.cc:302] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:32.774781 17867 raft_consensus.cc:2781] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:32.774901 17867 raft_consensus.cc:482] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:32.774996 17867 raft_consensus.cc:3037] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.776974 17867 raft_consensus.cc:504] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.777634 17867 leader_election.cc:288] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Requested vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:32.778061 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7553a4d675be4edb9c42f655b8e439da" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:32.778272 17602 raft_consensus.cc:3037] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.778206 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7553a4d675be4edb9c42f655b8e439da" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:32.778512 17062 raft_consensus.cc:3037] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.780396 17602 raft_consensus.cc:2454] T 7553a4d675be4edb9c42f655b8e439da P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:32.780809 17062 raft_consensus.cc:2454] T 7553a4d675be4edb9c42f655b8e439da P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:32.780828 17266 leader_election.cc:302] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:32.781149 17867 raft_consensus.cc:2781] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:32.781563 17867 raft_consensus.cc:686] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [term 1 LEADER]: Becoming Leader. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Running, Role: LEADER
I20240502 21:25:32.781877 17867 consensus_queue.cc:237] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.785055 16915 catalog_manager.cc:5484] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 967c7c03fe754179ab1544965af0cf8d (127.16.122.3). New cstate: current_term: 1 leader_uuid: "967c7c03fe754179ab1544965af0cf8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:32.805927 17867 raft_consensus.cc:482] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:32.806152 17867 raft_consensus.cc:504] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.806900 17867 leader_election.cc:288] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:32.807490 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08cd1c944e3f9a5438d204dfd196" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:32.807528 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08cd1c944e3f9a5438d204dfd196" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:32.807710 17602 raft_consensus.cc:2454] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:32.807828 17062 raft_consensus.cc:2454] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:32.808207 17266 leader_election.cc:302] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:32.808566 17867 raft_consensus.cc:2781] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:32.808666 17867 raft_consensus.cc:482] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:32.808775 17867 raft_consensus.cc:3037] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.810791 17867 raft_consensus.cc:504] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.811455 17867 leader_election.cc:288] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Requested vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:32.811864 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08cd1c944e3f9a5438d204dfd196" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:32.811995 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08cd1c944e3f9a5438d204dfd196" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:32.812119 17602 raft_consensus.cc:3037] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.812232 17062 raft_consensus.cc:3037] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.815109 17602 raft_consensus.cc:2454] T cbfe08cd1c944e3f9a5438d204dfd196 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:32.815107 17062 raft_consensus.cc:2454] T cbfe08cd1c944e3f9a5438d204dfd196 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:32.815621 17265 leader_election.cc:302] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:32.816195 17867 raft_consensus.cc:2781] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:32.816360 17867 raft_consensus.cc:686] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [term 1 LEADER]: Becoming Leader. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Running, Role: LEADER
I20240502 21:25:32.816629 17867 consensus_queue.cc:237] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:32.819733 16915 catalog_manager.cc:5484] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 967c7c03fe754179ab1544965af0cf8d (127.16.122.3). New cstate: current_term: 1 leader_uuid: "967c7c03fe754179ab1544965af0cf8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:32.915583 17843 raft_consensus.cc:482] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:32.915849 17843 raft_consensus.cc:504] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.916497 17843 leader_election.cc:288] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:32.917151 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5736b1f933416fb88223e3557cf57e" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:32.917188 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5736b1f933416fb88223e3557cf57e" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:32.917380 17197 raft_consensus.cc:2454] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:32.917438 17736 raft_consensus.cc:2454] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:32.917884 17401 leader_election.cc:302] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:32.918215 17843 raft_consensus.cc:2781] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:32.918323 17843 raft_consensus.cc:482] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:32.918480 17843 raft_consensus.cc:3037] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.920464 17843 raft_consensus.cc:504] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.920969 17843 leader_election.cc:288] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:32.921397 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5736b1f933416fb88223e3557cf57e" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:32.921428 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5736b1f933416fb88223e3557cf57e" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:32.921577 17197 raft_consensus.cc:3037] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.921614 17736 raft_consensus.cc:3037] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.923638 17197 raft_consensus.cc:2454] T ce5736b1f933416fb88223e3557cf57e P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:32.924049 17401 leader_election.cc:302] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:32.924083 17736 raft_consensus.cc:2454] T ce5736b1f933416fb88223e3557cf57e P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:32.924319 17843 raft_consensus.cc:2781] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:32.924829 17843 raft_consensus.cc:686] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:32.925144 17843 consensus_queue.cc:237] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:32.927954 16915 catalog_manager.cc:5484] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:33.297966 17867 consensus_queue.cc:1035] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.305739 17867 consensus_queue.cc:1035] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.316298 17843 consensus_queue.cc:1035] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.320322 18019 consensus_queue.cc:1035] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.338654 17843 consensus_queue.cc:1035] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.351893 18028 consensus_queue.cc:1035] T 7553a4d675be4edb9c42f655b8e439da P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:33.417520 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:33.417939 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:33.426720 18024 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 268.595 ms
time per row: 0.134298 ms
I20240502 21:25:33.858168 16915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:33.869588 17716 tablet_service.cc:1467] Processing CreateTablet for tablet e2d1434fc07c45f0be0a2015f8e31cae (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.869882 17448 tablet_service.cc:1467] Processing CreateTablet for tablet e2d1434fc07c45f0be0a2015f8e31cae (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.870141 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d1434fc07c45f0be0a2015f8e31cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.870558 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d1434fc07c45f0be0a2015f8e31cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.872282 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 2695b374d7fe417aa0a41ce5a06afa5d (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.872455 17175 tablet_service.cc:1467] Processing CreateTablet for tablet 2695b374d7fe417aa0a41ce5a06afa5d (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.872711 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2695b374d7fe417aa0a41ce5a06afa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.873046 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2695b374d7fe417aa0a41ce5a06afa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.875258 17313 tablet_service.cc:1467] Processing CreateTablet for tablet d684fc5640034615930845d0a908eb10 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.875749 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d684fc5640034615930845d0a908eb10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.875728 17176 tablet_service.cc:1467] Processing CreateTablet for tablet e2d1434fc07c45f0be0a2015f8e31cae (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.876228 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d1434fc07c45f0be0a2015f8e31cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.879375 17041 tablet_service.cc:1467] Processing CreateTablet for tablet d684fc5640034615930845d0a908eb10 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.879869 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d684fc5640034615930845d0a908eb10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.883335 17447 tablet_service.cc:1467] Processing CreateTablet for tablet 2695b374d7fe417aa0a41ce5a06afa5d (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.883339 17579 tablet_service.cc:1467] Processing CreateTablet for tablet d684fc5640034615930845d0a908eb10 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1e12577a03974c95adef526a704a980a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.883945 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2695b374d7fe417aa0a41ce5a06afa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.884040 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d684fc5640034615930845d0a908eb10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.898305 18087 tablet_bootstrap.cc:492] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:33.901085 18087 tablet_bootstrap.cc:654] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.901314 18089 tablet_bootstrap.cc:492] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:33.908170 18089 tablet_bootstrap.cc:654] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.910679 18093 tablet_bootstrap.cc:492] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:33.912281 18092 tablet_bootstrap.cc:492] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:33.915767 18090 tablet_bootstrap.cc:492] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:33.917551 18092 tablet_bootstrap.cc:654] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.919469 18093 tablet_bootstrap.cc:654] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.920369 18090 tablet_bootstrap.cc:654] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.922006 18087 tablet_bootstrap.cc:492] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:33.922209 18087 ts_tablet_manager.cc:1367] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.016s
I20240502 21:25:33.922212 18088 tablet_bootstrap.cc:492] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:33.923065 18092 tablet_bootstrap.cc:492] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:33.923239 18092 ts_tablet_manager.cc:1367] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240502 21:25:33.923213 18087 raft_consensus.cc:348] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:33.923496 18087 raft_consensus.cc:374] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.923609 18087 raft_consensus.cc:729] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.923888 18087 consensus_queue.cc:260] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:33.924425 18092 raft_consensus.cc:348] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:33.924639 18092 raft_consensus.cc:374] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.924724 18092 raft_consensus.cc:729] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.924950 18092 consensus_queue.cc:260] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:33.925729 18092 ts_tablet_manager.cc:1398] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:33.925861 18089 tablet_bootstrap.cc:492] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:33.926107 18089 ts_tablet_manager.cc:1367] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240502 21:25:33.927156 18089 raft_consensus.cc:348] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:33.927462 18089 raft_consensus.cc:374] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.927641 18089 raft_consensus.cc:729] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.928241 18088 tablet_bootstrap.cc:654] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.928122 18089 consensus_queue.cc:260] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:33.930711 18090 tablet_bootstrap.cc:492] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:33.930895 18090 ts_tablet_manager.cc:1367] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240502 21:25:33.931720 18089 ts_tablet_manager.cc:1398] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:25:33.931792 18090 raft_consensus.cc:348] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:33.931996 18090 raft_consensus.cc:374] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.932086 18090 raft_consensus.cc:729] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.932384 18090 consensus_queue.cc:260] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:33.933087 18090 ts_tablet_manager.cc:1398] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:33.934794 18089 tablet_bootstrap.cc:492] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:33.937006 18089 tablet_bootstrap.cc:654] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.940047 18093 tablet_bootstrap.cc:492] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:33.940335 18093 ts_tablet_manager.cc:1367] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20240502 21:25:33.941468 18093 raft_consensus.cc:348] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:33.941854 18093 raft_consensus.cc:374] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.942003 18093 raft_consensus.cc:729] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.942410 18093 consensus_queue.cc:260] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:33.943759 18087 ts_tablet_manager.cc:1398] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20240502 21:25:33.944278 18087 tablet_bootstrap.cc:492] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:33.946090 18088 tablet_bootstrap.cc:492] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:33.946159 18087 tablet_bootstrap.cc:654] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.946636 18093 ts_tablet_manager.cc:1398] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:25:33.947621 18088 ts_tablet_manager.cc:1367] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.012s
I20240502 21:25:33.948650 18088 raft_consensus.cc:348] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:33.948984 18088 raft_consensus.cc:374] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.949137 18088 raft_consensus.cc:729] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.949481 18088 consensus_queue.cc:260] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:33.953527 18088 ts_tablet_manager.cc:1398] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:25:33.956873 18088 tablet_bootstrap.cc:492] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:33.957917 18089 tablet_bootstrap.cc:492] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:33.958107 18089 ts_tablet_manager.cc:1367] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.007s
I20240502 21:25:33.958669 18088 tablet_bootstrap.cc:654] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.959009 18089 raft_consensus.cc:348] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:33.959268 18089 raft_consensus.cc:374] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.959359 18087 tablet_bootstrap.cc:492] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:33.959363 18089 raft_consensus.cc:729] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.959546 18087 ts_tablet_manager.cc:1367] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20240502 21:25:33.959646 18089 consensus_queue.cc:260] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:33.960379 18087 raft_consensus.cc:348] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:33.960609 18087 raft_consensus.cc:374] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.960695 18087 raft_consensus.cc:729] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.960956 18087 consensus_queue.cc:260] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:33.961160 18088 tablet_bootstrap.cc:492] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:33.961220 18089 ts_tablet_manager.cc:1398] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:33.961704 18087 ts_tablet_manager.cc:1398] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:25:33.962127 18088 ts_tablet_manager.cc:1367] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:25:33.963112 18088 raft_consensus.cc:348] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:33.963383 18088 raft_consensus.cc:374] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.963480 18088 raft_consensus.cc:729] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.963898 18088 consensus_queue.cc:260] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:33.966498 18088 ts_tablet_manager.cc:1398] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:34.057770 18097 raft_consensus.cc:482] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:34.057982 18097 raft_consensus.cc:504] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:34.059849 18097 leader_election.cc:288] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:34.071363 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2695b374d7fe417aa0a41ce5a06afa5d" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:34.071441 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2695b374d7fe417aa0a41ce5a06afa5d" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:34.071684 17468 raft_consensus.cc:2454] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:34.071713 17197 raft_consensus.cc:2454] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:34.072449 17668 leader_election.cc:302] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:34.073002 18097 raft_consensus.cc:2781] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:34.073194 18097 raft_consensus.cc:482] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:34.073321 18097 raft_consensus.cc:3037] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.073477 17925 raft_consensus.cc:482] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:34.073695 17925 raft_consensus.cc:504] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:34.074908 17925 leader_election.cc:288] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:34.075645 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1434fc07c45f0be0a2015f8e31cae" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:34.075997 17736 raft_consensus.cc:2454] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:34.076059 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1434fc07c45f0be0a2015f8e31cae" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:34.076294 17468 raft_consensus.cc:2454] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:34.076679 17129 leader_election.cc:302] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:34.077119 17925 raft_consensus.cc:2781] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:34.077226 17925 raft_consensus.cc:482] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:34.077438 17925 raft_consensus.cc:3037] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.172523 17925 raft_consensus.cc:504] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:34.172537 18097 raft_consensus.cc:504] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:34.173444 17925 leader_election.cc:288] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:34.173486 18097 leader_election.cc:288] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Requested vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:34.174150 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2695b374d7fe417aa0a41ce5a06afa5d" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:34.174432 17197 raft_consensus.cc:3037] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.174391 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1434fc07c45f0be0a2015f8e31cae" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:34.174497 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1434fc07c45f0be0a2015f8e31cae" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:34.174616 17467 raft_consensus.cc:3037] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.174744 17736 raft_consensus.cc:3037] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.177955 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2695b374d7fe417aa0a41ce5a06afa5d" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:34.178193 17468 raft_consensus.cc:3037] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.189880 18102 raft_consensus.cc:482] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:34.190199 18102 raft_consensus.cc:504] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:34.192337 18102 leader_election.cc:288] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:34.201037 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d684fc5640034615930845d0a908eb10" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:34.201366 17334 raft_consensus.cc:2454] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:34.201953 16995 leader_election.cc:302] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:34.202236 17601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d684fc5640034615930845d0a908eb10" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:34.202502 18102 raft_consensus.cc:2781] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:34.202569 17601 raft_consensus.cc:2454] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:34.202636 18102 raft_consensus.cc:482] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:34.202728 18102 raft_consensus.cc:3037] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.266863 17467 raft_consensus.cc:2454] T e2d1434fc07c45f0be0a2015f8e31cae P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:34.266855 17197 raft_consensus.cc:2454] T 2695b374d7fe417aa0a41ce5a06afa5d P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:34.266886 17736 raft_consensus.cc:2454] T e2d1434fc07c45f0be0a2015f8e31cae P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:34.267552 17129 leader_election.cc:302] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:34.267556 18102 raft_consensus.cc:504] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:34.268184 17925 raft_consensus.cc:2781] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:34.268433 18102 leader_election.cc:288] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Requested vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:34.268594 17668 leader_election.cc:302] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:34.269022 18097 raft_consensus.cc:2781] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:34.269981 17468 raft_consensus.cc:2454] T 2695b374d7fe417aa0a41ce5a06afa5d P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:34.270782 18097 raft_consensus.cc:686] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [term 1 LEADER]: Becoming Leader. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Running, Role: LEADER
I20240502 21:25:34.270882 17925 raft_consensus.cc:686] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:34.271494 18097 consensus_queue.cc:237] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:34.271569 17925 consensus_queue.cc:237] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:34.273808 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d684fc5640034615930845d0a908eb10" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:34.274056 17334 raft_consensus.cc:3037] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.276156 17601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d684fc5640034615930845d0a908eb10" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:34.276386 17601 raft_consensus.cc:3037] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:34.276850 16909 catalog_manager.cc:5484] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6). New cstate: current_term: 1 leader_uuid: "7181173c030d4a6db6e29c158fdb9a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:34.279866 16918 catalog_manager.cc:5484] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:34.367496 17334 raft_consensus.cc:2454] T d684fc5640034615930845d0a908eb10 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:34.367537 17601 raft_consensus.cc:2454] T d684fc5640034615930845d0a908eb10 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:34.368242 16995 leader_election.cc:302] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:34.368642 18102 raft_consensus.cc:2781] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:34.369086 18102 raft_consensus.cc:686] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Running, Role: LEADER
I20240502 21:25:34.369673 18102 consensus_queue.cc:237] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:34.373390 16918 catalog_manager.cc:5484] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "5a2cdeb7351f4fffb44146f4d9365403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:34.707944 17925 consensus_queue.cc:1035] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.714486 18121 consensus_queue.cc:1035] T e2d1434fc07c45f0be0a2015f8e31cae P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.748831 18097 consensus_queue.cc:1035] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.754482 18112 consensus_queue.cc:1035] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:34.830624 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:34.830999 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:34.839643 18117 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:34.839656 18102 consensus_queue.cc:1035] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:34.847301 18102 consensus_queue.cc:1035] T d684fc5640034615930845d0a908eb10 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 292.518 ms
time per row: 0.146259 ms
I20240502 21:25:35.295981 16917 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:35.306291 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 674644cf151b4f0f89f7b037263cc9a8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.306550 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 674644cf151b4f0f89f7b037263cc9a8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.306883 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674644cf151b4f0f89f7b037263cc9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.307005 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674644cf151b4f0f89f7b037263cc9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.307117 17175 tablet_service.cc:1467] Processing CreateTablet for tablet cd187707226242289cf3c2be45ccb955 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.307567 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd187707226242289cf3c2be45ccb955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.307660 17716 tablet_service.cc:1467] Processing CreateTablet for tablet cd187707226242289cf3c2be45ccb955 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.308105 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd187707226242289cf3c2be45ccb955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.311369 17313 tablet_service.cc:1467] Processing CreateTablet for tablet 674644cf151b4f0f89f7b037263cc9a8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.311807 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674644cf151b4f0f89f7b037263cc9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.323500 17447 tablet_service.cc:1467] Processing CreateTablet for tablet cd187707226242289cf3c2be45ccb955 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.323990 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd187707226242289cf3c2be45ccb955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.324258 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 52874dc477c0433984c9c59c583b8b20 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.324729 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52874dc477c0433984c9c59c583b8b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.325114 18188 tablet_bootstrap.cc:492] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:35.327240 18188 tablet_bootstrap.cc:654] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.327571 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 52874dc477c0433984c9c59c583b8b20 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.328022 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52874dc477c0433984c9c59c583b8b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.332988 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 52874dc477c0433984c9c59c583b8b20 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0ce557f3547484b8995efee7b5b9855]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.333628 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52874dc477c0433984c9c59c583b8b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.337440 18190 tablet_bootstrap.cc:492] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:35.338025 18193 tablet_bootstrap.cc:492] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:35.339015 18192 tablet_bootstrap.cc:492] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:35.340044 18193 tablet_bootstrap.cc:654] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.341907 18190 tablet_bootstrap.cc:654] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.342209 18186 tablet_bootstrap.cc:492] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:35.343416 18192 tablet_bootstrap.cc:654] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.343966 18188 tablet_bootstrap.cc:492] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:35.344552 18188 ts_tablet_manager.cc:1367] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20240502 21:25:35.345556 18188 raft_consensus.cc:348] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.345907 18186 tablet_bootstrap.cc:654] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.345918 18188 raft_consensus.cc:374] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.346028 18188 raft_consensus.cc:729] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.346554 18188 consensus_queue.cc:260] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.347290 18188 ts_tablet_manager.cc:1398] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:35.354158 18187 tablet_bootstrap.cc:492] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:35.354202 18193 tablet_bootstrap.cc:492] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:35.354259 18190 tablet_bootstrap.cc:492] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:35.354393 18193 ts_tablet_manager.cc:1367] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240502 21:25:35.354485 18190 ts_tablet_manager.cc:1367] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240502 21:25:35.355365 18190 raft_consensus.cc:348] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.355353 18193 raft_consensus.cc:348] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.355585 18193 raft_consensus.cc:374] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.355701 18190 raft_consensus.cc:374] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.355696 18193 raft_consensus.cc:729] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.355855 18190 raft_consensus.cc:729] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.355959 18193 consensus_queue.cc:260] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.356184 18190 consensus_queue.cc:260] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.356447 18187 tablet_bootstrap.cc:654] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.356957 18190 ts_tablet_manager.cc:1398] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:35.357436 18193 ts_tablet_manager.cc:1398] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:35.357743 18193 tablet_bootstrap.cc:492] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:35.359448 18193 tablet_bootstrap.cc:654] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.360324 18192 tablet_bootstrap.cc:492] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:35.360584 18192 ts_tablet_manager.cc:1367] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20240502 21:25:35.361519 18192 raft_consensus.cc:348] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.362092 18186 tablet_bootstrap.cc:492] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:35.361816 18192 raft_consensus.cc:374] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.362334 18186 ts_tablet_manager.cc:1367] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 21:25:35.362396 18192 raft_consensus.cc:729] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.362725 18192 consensus_queue.cc:260] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.363310 18186 raft_consensus.cc:348] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.363577 18186 raft_consensus.cc:374] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.363720 18186 raft_consensus.cc:729] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.364041 18186 consensus_queue.cc:260] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.365645 18186 ts_tablet_manager.cc:1398] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:35.366001 18186 tablet_bootstrap.cc:492] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:35.368904 18186 tablet_bootstrap.cc:654] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.368983 18192 ts_tablet_manager.cc:1398] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:25:35.374150 18186 tablet_bootstrap.cc:492] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:35.374315 18186 ts_tablet_manager.cc:1367] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 21:25:35.375165 18186 raft_consensus.cc:348] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.375334 18187 tablet_bootstrap.cc:492] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:35.375458 18186 raft_consensus.cc:374] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.375509 18187 ts_tablet_manager.cc:1367] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.015s
I20240502 21:25:35.375555 18186 raft_consensus.cc:729] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.375905 18193 tablet_bootstrap.cc:492] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:35.375850 18186 consensus_queue.cc:260] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.376091 18193 ts_tablet_manager.cc:1367] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240502 21:25:35.376230 18187 raft_consensus.cc:348] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.376399 18187 raft_consensus.cc:374] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.376523 18187 raft_consensus.cc:729] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.376542 18186 ts_tablet_manager.cc:1398] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:35.377439 18193 raft_consensus.cc:348] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.377650 18193 raft_consensus.cc:374] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.377736 18193 raft_consensus.cc:729] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.377985 18193 consensus_queue.cc:260] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.379463 18193 ts_tablet_manager.cc:1398] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:35.376754 18187 consensus_queue.cc:260] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.381855 18187 ts_tablet_manager.cc:1398] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:25:35.382207 18187 tablet_bootstrap.cc:492] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:35.384013 18187 tablet_bootstrap.cc:654] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.386107 18187 tablet_bootstrap.cc:492] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:35.386332 18187 ts_tablet_manager.cc:1367] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:35.387010 18187 raft_consensus.cc:348] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.387197 18187 raft_consensus.cc:374] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.387279 18187 raft_consensus.cc:729] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.387490 18187 consensus_queue.cc:260] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.387895 18187 ts_tablet_manager.cc:1398] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:25:35.417515 18028 raft_consensus.cc:482] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:35.417783 18028 raft_consensus.cc:504] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.418632 18028 leader_election.cc:288] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:35.419276 17601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674644cf151b4f0f89f7b037263cc9a8" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:35.419299 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674644cf151b4f0f89f7b037263cc9a8" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:35.419550 17061 raft_consensus.cc:2454] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:35.419575 17601 raft_consensus.cc:2454] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:35.420163 17265 leader_election.cc:302] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:35.420539 18028 raft_consensus.cc:2781] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:35.420632 18028 raft_consensus.cc:482] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:35.420758 18028 raft_consensus.cc:3037] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.445035 18121 raft_consensus.cc:482] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:35.445287 18121 raft_consensus.cc:504] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.446028 18121 leader_election.cc:288] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:35.446727 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd187707226242289cf3c2be45ccb955" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:35.446728 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd187707226242289cf3c2be45ccb955" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:35.446979 17736 raft_consensus.cc:2454] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:35.446980 17467 raft_consensus.cc:2454] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:35.447463 17129 leader_election.cc:302] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:35.447759 18121 raft_consensus.cc:2781] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:35.447861 18121 raft_consensus.cc:482] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:35.447978 18121 raft_consensus.cc:3037] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.481526 18121 raft_consensus.cc:504] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.481566 18028 raft_consensus.cc:504] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.482748 18028 leader_election.cc:288] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Requested vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:35.483001 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd187707226242289cf3c2be45ccb955" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:35.483002 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd187707226242289cf3c2be45ccb955" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:35.483273 17736 raft_consensus.cc:3037] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.483274 17467 raft_consensus.cc:3037] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.483464 17601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674644cf151b4f0f89f7b037263cc9a8" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:35.483714 17601 raft_consensus.cc:3037] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.486145 17467 raft_consensus.cc:2454] T cd187707226242289cf3c2be45ccb955 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:35.486618 17736 raft_consensus.cc:2454] T cd187707226242289cf3c2be45ccb955 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:35.487200 17129 leader_election.cc:302] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:35.487550 17925 raft_consensus.cc:2781] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:35.487715 17925 raft_consensus.cc:686] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:35.487815 18121 leader_election.cc:288] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:35.488009 17925 consensus_queue.cc:237] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:35.488519 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674644cf151b4f0f89f7b037263cc9a8" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:35.488744 17061 raft_consensus.cc:3037] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.491715 16912 catalog_manager.cc:5484] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:35.584345 17601 raft_consensus.cc:2454] T 674644cf151b4f0f89f7b037263cc9a8 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:35.584370 17061 raft_consensus.cc:2454] T 674644cf151b4f0f89f7b037263cc9a8 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:35.585073 17265 leader_election.cc:302] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:35.585512 18028 raft_consensus.cc:2781] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:35.586035 18028 raft_consensus.cc:686] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [term 1 LEADER]: Becoming Leader. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Running, Role: LEADER
I20240502 21:25:35.586445 18028 consensus_queue.cc:237] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:35.591447 16912 catalog_manager.cc:5484] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 967c7c03fe754179ab1544965af0cf8d (127.16.122.3). New cstate: current_term: 1 leader_uuid: "967c7c03fe754179ab1544965af0cf8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:35.618165 18115 raft_consensus.cc:482] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:35.618394 18115 raft_consensus.cc:504] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.619700 18115 leader_election.cc:288] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:35.620268 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52874dc477c0433984c9c59c583b8b20" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:35.620265 17601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52874dc477c0433984c9c59c583b8b20" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:35.620532 17601 raft_consensus.cc:2454] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:35.620518 17334 raft_consensus.cc:2454] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:35.620973 16996 leader_election.cc:302] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:35.621263 18115 raft_consensus.cc:2781] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:35.621403 18115 raft_consensus.cc:482] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:35.621480 18115 raft_consensus.cc:3037] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.687244 18115 raft_consensus.cc:504] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.688067 18115 leader_election.cc:288] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Requested vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:35.688748 17601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52874dc477c0433984c9c59c583b8b20" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:35.688768 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52874dc477c0433984c9c59c583b8b20" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:35.688958 17601 raft_consensus.cc:3037] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.689035 17334 raft_consensus.cc:3037] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.788542 17334 raft_consensus.cc:2454] T 52874dc477c0433984c9c59c583b8b20 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:35.788542 17601 raft_consensus.cc:2454] T 52874dc477c0433984c9c59c583b8b20 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:35.789461 16996 leader_election.cc:302] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:35.789968 18115 raft_consensus.cc:2781] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:35.790190 18115 raft_consensus.cc:686] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Running, Role: LEADER
I20240502 21:25:35.790583 18115 consensus_queue.cc:237] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:35.793603 16912 catalog_manager.cc:5484] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "5a2cdeb7351f4fffb44146f4d9365403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:35.990227 17925 consensus_queue.cc:1035] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.010448 18211 consensus_queue.cc:1035] T cd187707226242289cf3c2be45ccb955 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.090335 18028 consensus_queue.cc:1035] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.114509 18215 consensus_queue.cc:1035] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.175233 18115 consensus_queue.cc:1035] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:36.182631 18115 consensus_queue.cc:1035] T 52874dc477c0433984c9c59c583b8b20 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:36.314939 18206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.315335 18206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.323732 18206 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 264.898 ms
time per row: 0.132449 ms
I20240502 21:25:36.753667 16915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:36.764822 17716 tablet_service.cc:1467] Processing CreateTablet for tablet c7115fd506c540eaa3ad9c3904c410e8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.765244 17447 tablet_service.cc:1467] Processing CreateTablet for tablet c7115fd506c540eaa3ad9c3904c410e8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.765376 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7115fd506c540eaa3ad9c3904c410e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.765686 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7115fd506c540eaa3ad9c3904c410e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.766085 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 0070b68a1e5242e5a71ac1c7433c0425 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.766681 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0070b68a1e5242e5a71ac1c7433c0425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.768180 17175 tablet_service.cc:1467] Processing CreateTablet for tablet c7115fd506c540eaa3ad9c3904c410e8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.768651 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7115fd506c540eaa3ad9c3904c410e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.770081 17176 tablet_service.cc:1467] Processing CreateTablet for tablet c839ab1b86104fe6a1b6fbf7ddff1d19 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.770498 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c839ab1b86104fe6a1b6fbf7ddff1d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.770884 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 0070b68a1e5242e5a71ac1c7433c0425 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.771322 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0070b68a1e5242e5a71ac1c7433c0425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.771274 17715 tablet_service.cc:1467] Processing CreateTablet for tablet c839ab1b86104fe6a1b6fbf7ddff1d19 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.771713 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c839ab1b86104fe6a1b6fbf7ddff1d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.779083 17313 tablet_service.cc:1467] Processing CreateTablet for tablet 0070b68a1e5242e5a71ac1c7433c0425 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.779635 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0070b68a1e5242e5a71ac1c7433c0425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.783483 17448 tablet_service.cc:1467] Processing CreateTablet for tablet c839ab1b86104fe6a1b6fbf7ddff1d19 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e494795229d34acfba91c5f05eaaa568]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:36.784106 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c839ab1b86104fe6a1b6fbf7ddff1d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.791496 18281 tablet_bootstrap.cc:492] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:36.792325 18280 tablet_bootstrap.cc:492] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:36.794788 18280 tablet_bootstrap.cc:654] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.795207 18281 tablet_bootstrap.cc:654] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.796959 18284 tablet_bootstrap.cc:492] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:36.800567 18281 tablet_bootstrap.cc:492] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:36.800853 18281 ts_tablet_manager.cc:1367] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:25:36.802137 18281 raft_consensus.cc:348] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.802559 18281 raft_consensus.cc:374] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.802721 18281 raft_consensus.cc:729] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.803073 18281 consensus_queue.cc:260] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.803929 18281 ts_tablet_manager.cc:1398] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:36.808626 18284 tablet_bootstrap.cc:654] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.813647 17906 raft_consensus.cc:482] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:36.813825 17906 raft_consensus.cc:504] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.814635 17906 leader_election.cc:288] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:36.815786 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0070b68a1e5242e5a71ac1c7433c0425" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:36.816370 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0070b68a1e5242e5a71ac1c7433c0425" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
W20240502 21:25:36.817085 17534 leader_election.cc:341] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:25:36.817571 17534 leader_election.cc:341] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:36.817703 17534 leader_election.cc:302] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238; no voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d
I20240502 21:25:36.818018 18282 tablet_bootstrap.cc:492] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:36.818255 17906 raft_consensus.cc:2726] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:36.819317 18288 tablet_bootstrap.cc:492] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:36.820372 18282 tablet_bootstrap.cc:654] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.821398 18288 tablet_bootstrap.cc:654] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.822132 18284 tablet_bootstrap.cc:492] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:36.822603 18283 tablet_bootstrap.cc:492] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:36.822396 18284 ts_tablet_manager.cc:1367] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240502 21:25:36.823791 18280 tablet_bootstrap.cc:492] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:36.824029 18280 ts_tablet_manager.cc:1367] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.003s
I20240502 21:25:36.823974 18284 raft_consensus.cc:348] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.824270 18284 raft_consensus.cc:374] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.824441 18284 raft_consensus.cc:729] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.824805 18284 consensus_queue.cc:260] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.824935 18283 tablet_bootstrap.cc:654] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.825110 18280 raft_consensus.cc:348] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.825387 18280 raft_consensus.cc:374] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.825532 18280 raft_consensus.cc:729] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.825855 18280 consensus_queue.cc:260] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.829106 18284 ts_tablet_manager.cc:1398] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:25:36.829478 18284 tablet_bootstrap.cc:492] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:36.831293 18284 tablet_bootstrap.cc:654] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.835309 18280 ts_tablet_manager.cc:1398] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 21:25:36.836700 18280 tablet_bootstrap.cc:492] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:36.838769 18280 tablet_bootstrap.cc:654] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.841367 18288 tablet_bootstrap.cc:492] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:36.841650 18288 ts_tablet_manager.cc:1367] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 21:25:36.842974 18288 raft_consensus.cc:348] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.843222 18288 raft_consensus.cc:374] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.843370 18288 raft_consensus.cc:729] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.843667 18288 consensus_queue.cc:260] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.844420 18288 ts_tablet_manager.cc:1398] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:36.845958 18283 tablet_bootstrap.cc:492] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:36.846177 18283 ts_tablet_manager.cc:1367] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.013s
I20240502 21:25:36.846818 18282 tablet_bootstrap.cc:492] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:36.846992 18282 ts_tablet_manager.cc:1367] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.018s
I20240502 21:25:36.847311 18283 raft_consensus.cc:348] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.847601 18283 raft_consensus.cc:374] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.847754 18283 raft_consensus.cc:729] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.848011 18282 raft_consensus.cc:348] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.848232 18282 raft_consensus.cc:374] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.848335 18282 raft_consensus.cc:729] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.848091 18283 consensus_queue.cc:260] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:36.848558 18282 consensus_queue.cc:260] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.849090 18282 ts_tablet_manager.cc:1398] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:36.849408 18282 tablet_bootstrap.cc:492] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:36.851315 18283 ts_tablet_manager.cc:1398] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:25:36.851740 18284 tablet_bootstrap.cc:492] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:36.851934 18284 ts_tablet_manager.cc:1367] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.005s
I20240502 21:25:36.852214 18282 tablet_bootstrap.cc:654] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.852763 18284 raft_consensus.cc:348] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.853008 18284 raft_consensus.cc:374] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.853086 18284 raft_consensus.cc:729] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.854056 18280 tablet_bootstrap.cc:492] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:36.854240 18280 ts_tablet_manager.cc:1367] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 21:25:36.855110 18280 raft_consensus.cc:348] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.855329 18280 raft_consensus.cc:374] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.855404 18280 raft_consensus.cc:729] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.855646 18280 consensus_queue.cc:260] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.856317 18280 ts_tablet_manager.cc:1398] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:36.856549 18284 consensus_queue.cc:260] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.857160 18284 ts_tablet_manager.cc:1398] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:25:36.860610 18282 tablet_bootstrap.cc:492] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:36.860833 18282 ts_tablet_manager.cc:1367] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 21:25:36.861685 18282 raft_consensus.cc:348] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.861943 18282 raft_consensus.cc:374] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.862038 18282 raft_consensus.cc:729] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.862408 18282 consensus_queue.cc:260] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.863088 18282 ts_tablet_manager.cc:1398] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:36.979799 18112 raft_consensus.cc:482] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:36.980086 18112 raft_consensus.cc:504] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:36.980863 18112 leader_election.cc:288] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:36.981508 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839ab1b86104fe6a1b6fbf7ddff1d19" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:36.981518 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839ab1b86104fe6a1b6fbf7ddff1d19" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:36.981743 17467 raft_consensus.cc:2454] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:36.981766 17197 raft_consensus.cc:2454] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:36.982285 17670 leader_election.cc:302] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:36.982635 18112 raft_consensus.cc:2781] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:36.982738 18112 raft_consensus.cc:482] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:36.982857 18112 raft_consensus.cc:3037] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:36.990615 17843 raft_consensus.cc:482] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:36.990814 17843 raft_consensus.cc:504] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:36.991571 17843 leader_election.cc:288] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:36.992128 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7115fd506c540eaa3ad9c3904c410e8" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:36.992158 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7115fd506c540eaa3ad9c3904c410e8" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:36.992357 17197 raft_consensus.cc:2454] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:36.992355 17736 raft_consensus.cc:2454] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:36.992820 17401 leader_election.cc:302] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:36.993084 17843 raft_consensus.cc:2781] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:36.993253 17843 raft_consensus.cc:482] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:36.993355 17843 raft_consensus.cc:3037] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.015075 18112 raft_consensus.cc:504] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:37.015105 17843 raft_consensus.cc:504] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:37.015952 17843 leader_election.cc:288] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:37.016625 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839ab1b86104fe6a1b6fbf7ddff1d19" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:37.016872 17197 raft_consensus.cc:3037] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.016789 17196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7115fd506c540eaa3ad9c3904c410e8" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:37.017135 17196 raft_consensus.cc:3037] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.017155 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7115fd506c540eaa3ad9c3904c410e8" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:37.017351 17736 raft_consensus.cc:3037] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.020254 17736 raft_consensus.cc:2454] T c7115fd506c540eaa3ad9c3904c410e8 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:37.020262 17196 raft_consensus.cc:2454] T c7115fd506c540eaa3ad9c3904c410e8 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:37.020344 17197 raft_consensus.cc:2454] T c839ab1b86104fe6a1b6fbf7ddff1d19 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:37.020742 18112 leader_election.cc:288] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Requested vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:37.020840 17668 leader_election.cc:302] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:37.021147 17401 leader_election.cc:302] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:37.021217 18112 raft_consensus.cc:2781] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:37.021636 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839ab1b86104fe6a1b6fbf7ddff1d19" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:37.021878 17467 raft_consensus.cc:3037] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.021927 17843 raft_consensus.cc:2781] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:37.023191 18112 raft_consensus.cc:686] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 LEADER]: Becoming Leader. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Running, Role: LEADER
I20240502 21:25:37.023665 17843 raft_consensus.cc:686] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:37.024032 18112 consensus_queue.cc:237] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:37.024084 17843 consensus_queue.cc:237] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:37.027464 16914 catalog_manager.cc:5484] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:37.027417 16912 catalog_manager.cc:5484] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6). New cstate: current_term: 1 leader_uuid: "7181173c030d4a6db6e29c158fdb9a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:37.125883 18262 raft_consensus.cc:482] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:37.126093 18262 raft_consensus.cc:504] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:37.126768 18262 leader_election.cc:288] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:37.127302 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0070b68a1e5242e5a71ac1c7433c0425" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:37.127351 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0070b68a1e5242e5a71ac1c7433c0425" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:37.127571 17602 raft_consensus.cc:2454] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:37.127586 17334 raft_consensus.cc:2454] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:37.128036 16995 leader_election.cc:302] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:37.128346 18262 raft_consensus.cc:2781] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:37.128466 18262 raft_consensus.cc:482] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:37.128549 18262 raft_consensus.cc:3037] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.143882 17467 raft_consensus.cc:2454] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:37.146119 18262 raft_consensus.cc:504] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:37.147454 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0070b68a1e5242e5a71ac1c7433c0425" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:37.147683 17602 raft_consensus.cc:3037] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.147644 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0070b68a1e5242e5a71ac1c7433c0425" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:37.147861 17334 raft_consensus.cc:3037] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:37.147007 18262 leader_election.cc:288] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Requested vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:37.242449 17334 raft_consensus.cc:2454] T 0070b68a1e5242e5a71ac1c7433c0425 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:37.242437 17602 raft_consensus.cc:2454] T 0070b68a1e5242e5a71ac1c7433c0425 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:37.243111 16996 leader_election.cc:302] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:37.243532 18262 raft_consensus.cc:2781] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:37.243690 18262 raft_consensus.cc:686] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Running, Role: LEADER
I20240502 21:25:37.243997 18262 consensus_queue.cc:237] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:37.247012 16914 catalog_manager.cc:5484] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "5a2cdeb7351f4fffb44146f4d9365403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:37.542212 18112 consensus_queue.cc:1035] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:37.551571 18304 consensus_queue.cc:1035] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:37.636319 17843 consensus_queue.cc:1035] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:37.639405 17843 consensus_queue.cc:1035] T c7115fd506c540eaa3ad9c3904c410e8 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:37.662891 18302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.663237 18302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:37.672113 18302 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:25:37.759745 18262 consensus_queue.cc:1035] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:37.797953 18262 consensus_queue.cc:1035] T 0070b68a1e5242e5a71ac1c7433c0425 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 333.295 ms
time per row: 0.166647 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:39.171057 16911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:39.181413 17313 tablet_service.cc:1467] Processing CreateTablet for tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.181658 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.181874 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.182148 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.183187 17448 tablet_service.cc:1467] Processing CreateTablet for tablet 125cd0bebf43445f8f2d1d7a54f7c036 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.183280 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.183813 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125cd0bebf43445f8f2d1d7a54f7c036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.183827 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.188587 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 125cd0bebf43445f8f2d1d7a54f7c036 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.188583 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 36b6b065fc0c49cea3fac5df0fbdf096 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.188596 17176 tablet_service.cc:1467] Processing CreateTablet for tablet 125cd0bebf43445f8f2d1d7a54f7c036 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.189041 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125cd0bebf43445f8f2d1d7a54f7c036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.189049 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b6b065fc0c49cea3fac5df0fbdf096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.189239 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125cd0bebf43445f8f2d1d7a54f7c036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.190866 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 36b6b065fc0c49cea3fac5df0fbdf096 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.191442 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b6b065fc0c49cea3fac5df0fbdf096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.194824 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 36b6b065fc0c49cea3fac5df0fbdf096 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fcfea677a524de486afb50ad2b0e504]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.195283 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b6b065fc0c49cea3fac5df0fbdf096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.199752 18383 tablet_bootstrap.cc:492] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:39.200397 18382 tablet_bootstrap.cc:492] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:39.201592 18383 tablet_bootstrap.cc:654] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.202059 18382 tablet_bootstrap.cc:654] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.210163 18387 tablet_bootstrap.cc:492] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:39.211637 18386 tablet_bootstrap.cc:492] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:39.214197 18386 tablet_bootstrap.cc:654] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.214197 18387 tablet_bootstrap.cc:654] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.217329 18382 tablet_bootstrap.cc:492] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:39.217614 18382 ts_tablet_manager.cc:1367] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240502 21:25:39.217669 18387 tablet_bootstrap.cc:492] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:39.217809 18387 ts_tablet_manager.cc:1367] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:25:39.218502 18390 tablet_bootstrap.cc:492] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:39.218633 18387 raft_consensus.cc:348] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.218616 18382 raft_consensus.cc:348] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.218850 18382 raft_consensus.cc:374] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.218936 18382 raft_consensus.cc:729] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.219297 18382 consensus_queue.cc:260] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.220239 18390 tablet_bootstrap.cc:654] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.220791 18387 raft_consensus.cc:374] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.220842 18382 ts_tablet_manager.cc:1398] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:39.224848 18382 tablet_bootstrap.cc:492] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:39.225407 18387 raft_consensus.cc:729] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.228662 18382 tablet_bootstrap.cc:654] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.226080 18387 consensus_queue.cc:260] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.230919 18387 ts_tablet_manager.cc:1398] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:25:39.231104 18390 tablet_bootstrap.cc:492] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:39.231369 18390 ts_tablet_manager.cc:1367] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240502 21:25:39.232496 18390 raft_consensus.cc:348] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.232944 18390 raft_consensus.cc:374] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.233093 18390 raft_consensus.cc:729] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.235971 18390 consensus_queue.cc:260] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.237905 18390 ts_tablet_manager.cc:1398] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:25:39.239295 18391 tablet_bootstrap.cc:492] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:39.239521 18382 tablet_bootstrap.cc:492] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:39.239812 18382 ts_tablet_manager.cc:1367] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20240502 21:25:39.240754 18382 raft_consensus.cc:348] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.241060 18382 raft_consensus.cc:374] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.241195 18382 raft_consensus.cc:729] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.241402 18391 tablet_bootstrap.cc:654] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.241624 18382 consensus_queue.cc:260] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.242311 18382 ts_tablet_manager.cc:1398] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:39.244220 18386 tablet_bootstrap.cc:492] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:39.244477 18386 ts_tablet_manager.cc:1367] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.011s
I20240502 21:25:39.245591 18386 raft_consensus.cc:348] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.245755 18383 tablet_bootstrap.cc:492] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:39.245916 18391 tablet_bootstrap.cc:492] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:39.245880 18386 raft_consensus.cc:374] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.245950 18383 ts_tablet_manager.cc:1367] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.000s
I20240502 21:25:39.246055 18386 raft_consensus.cc:729] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.246121 18391 ts_tablet_manager.cc:1367] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 21:25:39.246451 18386 consensus_queue.cc:260] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.246809 18383 raft_consensus.cc:348] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.247057 18383 raft_consensus.cc:374] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.246958 18391 raft_consensus.cc:348] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.247166 18383 raft_consensus.cc:729] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.247184 18391 raft_consensus.cc:374] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.247316 18391 raft_consensus.cc:729] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.247546 18391 consensus_queue.cc:260] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.248081 18391 ts_tablet_manager.cc:1398] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:39.248162 18383 consensus_queue.cc:260] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.249014 18386 ts_tablet_manager.cc:1398] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:39.249387 18383 ts_tablet_manager.cc:1398] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:39.249612 18386 tablet_bootstrap.cc:492] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:39.249701 18383 tablet_bootstrap.cc:492] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:39.251498 18386 tablet_bootstrap.cc:654] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.251677 18383 tablet_bootstrap.cc:654] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.259480 18383 tablet_bootstrap.cc:492] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:39.259675 18383 ts_tablet_manager.cc:1367] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:25:39.260484 18383 raft_consensus.cc:348] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.260789 18383 raft_consensus.cc:374] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.260895 18383 raft_consensus.cc:729] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.261173 18383 consensus_queue.cc:260] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.261767 18383 ts_tablet_manager.cc:1398] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:39.264349 18386 tablet_bootstrap.cc:492] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:39.264519 18386 ts_tablet_manager.cc:1367] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20240502 21:25:39.265295 18386 raft_consensus.cc:348] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.265480 18386 raft_consensus.cc:374] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.265550 18386 raft_consensus.cc:729] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.265779 18386 consensus_queue.cc:260] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.266292 18386 ts_tablet_manager.cc:1398] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:39.322543 17925 raft_consensus.cc:482] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:39.322790 17925 raft_consensus.cc:504] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.323511 17925 leader_election.cc:288] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:39.324107 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125cd0bebf43445f8f2d1d7a54f7c036" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:39.324440 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125cd0bebf43445f8f2d1d7a54f7c036" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:39.324542 17736 raft_consensus.cc:2454] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:39.324712 17468 raft_consensus.cc:2454] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:39.325104 17129 leader_election.cc:302] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:39.325453 17925 raft_consensus.cc:2781] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:39.325548 17925 raft_consensus.cc:482] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:39.325655 17925 raft_consensus.cc:3037] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.327636 17925 raft_consensus.cc:504] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.328236 17925 leader_election.cc:288] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:39.328745 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125cd0bebf43445f8f2d1d7a54f7c036" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:39.328779 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125cd0bebf43445f8f2d1d7a54f7c036" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:39.328950 17468 raft_consensus.cc:3037] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.328976 17736 raft_consensus.cc:3037] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.330874 17468 raft_consensus.cc:2454] T 125cd0bebf43445f8f2d1d7a54f7c036 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:39.330896 17736 raft_consensus.cc:2454] T 125cd0bebf43445f8f2d1d7a54f7c036 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:39.331270 17131 leader_election.cc:302] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:39.331655 17925 raft_consensus.cc:2781] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.332091 17925 raft_consensus.cc:686] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:39.332407 17925 consensus_queue.cc:237] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } }
I20240502 21:25:39.335950 16917 catalog_manager.cc:5484] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:39.389271 17906 raft_consensus.cc:482] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:39.389469 17906 raft_consensus.cc:504] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.390120 17906 leader_election.cc:288] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:39.390815 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb4c9d1de94ec6b8e1ab4eec11af9f" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:39.390868 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb4c9d1de94ec6b8e1ab4eec11af9f" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:39.391077 17334 raft_consensus.cc:2454] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:39.391080 17062 raft_consensus.cc:2454] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:39.391559 17534 leader_election.cc:302] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:39.391840 17906 raft_consensus.cc:2781] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:39.391983 17906 raft_consensus.cc:482] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:39.392083 17906 raft_consensus.cc:3037] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.394001 17906 raft_consensus.cc:504] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.394595 17906 leader_election.cc:288] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:39.395048 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb4c9d1de94ec6b8e1ab4eec11af9f" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:39.395131 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb4c9d1de94ec6b8e1ab4eec11af9f" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:39.395267 17334 raft_consensus.cc:3037] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.395363 17062 raft_consensus.cc:3037] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.397459 17334 raft_consensus.cc:2454] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:39.397457 17062 raft_consensus.cc:2454] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:39.397876 17534 leader_election.cc:302] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:39.398195 17906 raft_consensus.cc:2781] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.398723 17906 raft_consensus.cc:686] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:39.399061 17906 consensus_queue.cc:237] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.402419 16917 catalog_manager.cc:5484] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:39.528776 17906 raft_consensus.cc:482] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:39.529062 17906 raft_consensus.cc:504] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.529789 17906 leader_election.cc:288] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:39.530687 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b6b065fc0c49cea3fac5df0fbdf096" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:39.530661 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b6b065fc0c49cea3fac5df0fbdf096" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:39.530938 17334 raft_consensus.cc:2454] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:39.530982 17062 raft_consensus.cc:2454] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:39.531445 17534 leader_election.cc:302] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:39.531741 17906 raft_consensus.cc:2781] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:39.531841 17906 raft_consensus.cc:482] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:39.531957 17906 raft_consensus.cc:3037] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.534000 17906 raft_consensus.cc:504] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.534607 17906 leader_election.cc:288] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:39.535229 17334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b6b065fc0c49cea3fac5df0fbdf096" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:39.535497 17334 raft_consensus.cc:3037] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.535493 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b6b065fc0c49cea3fac5df0fbdf096" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:39.535750 17062 raft_consensus.cc:3037] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.538666 17062 raft_consensus.cc:2454] T 36b6b065fc0c49cea3fac5df0fbdf096 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:39.538689 17334 raft_consensus.cc:2454] T 36b6b065fc0c49cea3fac5df0fbdf096 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:39.539444 17534 leader_election.cc:302] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:39.539817 17906 raft_consensus.cc:2781] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.539988 17906 raft_consensus.cc:686] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:39.540271 17906 consensus_queue.cc:237] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:39.543356 16917 catalog_manager.cc:5484] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:39.817150 17906 consensus_queue.cc:1035] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:39.824585 18399 consensus_queue.cc:1035] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:39.858392 18397 consensus_queue.cc:1035] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:39.866919 18397 consensus_queue.cc:1035] T 125cd0bebf43445f8f2d1d7a54f7c036 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:39.947062 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.947425 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.955155 18402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 21:25:40.079828 17906 consensus_queue.cc:1035] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:40.081043 17906 consensus_queue.cc:1035] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 261.231 ms
time per row: 0.130616 ms
I20240502 21:25:40.372455 16915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:40.382481 17448 tablet_service.cc:1467] Processing CreateTablet for tablet 8faf03c730e341b8bc3edb9a72d03d39 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.382652 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 8faf03c730e341b8bc3edb9a72d03d39 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.383035 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8faf03c730e341b8bc3edb9a72d03d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.383136 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8faf03c730e341b8bc3edb9a72d03d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.384312 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbccee0bdc9476c99e9926db224c5b9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.384645 17176 tablet_service.cc:1467] Processing CreateTablet for tablet 8faf03c730e341b8bc3edb9a72d03d39 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.384773 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbccee0bdc9476c99e9926db224c5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.385099 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8faf03c730e341b8bc3edb9a72d03d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.387424 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5df765230c4f45bbd53d2e1e603aab (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.387920 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5df765230c4f45bbd53d2e1e603aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.388590 17175 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbccee0bdc9476c99e9926db224c5b9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.389041 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbccee0bdc9476c99e9926db224c5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.390759 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5df765230c4f45bbd53d2e1e603aab (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.391268 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5df765230c4f45bbd53d2e1e603aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.395216 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5df765230c4f45bbd53d2e1e603aab (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.395743 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5df765230c4f45bbd53d2e1e603aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.399349 17447 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbccee0bdc9476c99e9926db224c5b9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b101975971bf4f53bc6bfe8adece0c67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:40.399858 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbccee0bdc9476c99e9926db224c5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:40.400069 18473 tablet_bootstrap.cc:492] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:40.402576 18473 tablet_bootstrap.cc:654] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.412573 18473 tablet_bootstrap.cc:492] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:40.413594 18473 ts_tablet_manager.cc:1367] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20240502 21:25:40.414873 18473 raft_consensus.cc:348] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.415256 18473 raft_consensus.cc:374] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.415413 18473 raft_consensus.cc:729] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.415819 18473 consensus_queue.cc:260] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.421489 18473 ts_tablet_manager.cc:1398] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:25:40.426043 18473 tablet_bootstrap.cc:492] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:40.429637 18473 tablet_bootstrap.cc:654] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.443027 18476 tablet_bootstrap.cc:492] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:40.445319 18473 tablet_bootstrap.cc:492] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:40.445676 18473 ts_tablet_manager.cc:1367] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20240502 21:25:40.445773 18476 tablet_bootstrap.cc:654] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.446852 18473 raft_consensus.cc:348] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.447652 18473 raft_consensus.cc:374] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.447811 18479 tablet_bootstrap.cc:492] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:40.447813 18473 raft_consensus.cc:729] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.448241 18477 tablet_bootstrap.cc:492] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:40.448201 18473 consensus_queue.cc:260] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.450824 18477 tablet_bootstrap.cc:654] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.453414 18479 tablet_bootstrap.cc:654] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.459031 18473 ts_tablet_manager.cc:1398] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:25:40.464517 18479 tablet_bootstrap.cc:492] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:40.465718 18476 tablet_bootstrap.cc:492] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:40.465982 18476 ts_tablet_manager.cc:1367] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.006s
I20240502 21:25:40.466671 18479 ts_tablet_manager.cc:1367] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240502 21:25:40.467196 18476 raft_consensus.cc:348] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.468174 18476 raft_consensus.cc:374] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.468392 18476 raft_consensus.cc:729] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.468714 18474 tablet_bootstrap.cc:492] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:40.468753 18476 consensus_queue.cc:260] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.469700 18479 raft_consensus.cc:348] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.469981 18479 raft_consensus.cc:374] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.470181 18472 tablet_bootstrap.cc:492] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:40.470266 18476 ts_tablet_manager.cc:1398] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:40.470826 18477 tablet_bootstrap.cc:492] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:40.470992 18477 ts_tablet_manager.cc:1367] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20240502 21:25:40.471879 18477 raft_consensus.cc:348] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.472139 18477 raft_consensus.cc:374] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.472157 18472 tablet_bootstrap.cc:654] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.472244 18477 raft_consensus.cc:729] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.472534 18477 consensus_queue.cc:260] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.472851 18479 raft_consensus.cc:729] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.473156 18479 consensus_queue.cc:260] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.473811 18479 ts_tablet_manager.cc:1398] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 21:25:40.474401 18477 ts_tablet_manager.cc:1398] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:40.476658 18474 tablet_bootstrap.cc:654] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.479569 18474 tablet_bootstrap.cc:492] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:40.479573 18472 tablet_bootstrap.cc:492] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:40.479748 18472 ts_tablet_manager.cc:1367] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:25:40.479827 18474 ts_tablet_manager.cc:1367] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 21:25:40.480720 18472 raft_consensus.cc:348] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.480720 18474 raft_consensus.cc:348] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.480939 18474 raft_consensus.cc:374] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.481038 18472 raft_consensus.cc:374] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.481043 18474 raft_consensus.cc:729] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.481150 18472 raft_consensus.cc:729] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.481300 18474 consensus_queue.cc:260] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.481408 18472 consensus_queue.cc:260] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.481904 18474 ts_tablet_manager.cc:1398] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:40.482283 18474 tablet_bootstrap.cc:492] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:40.483153 18472 ts_tablet_manager.cc:1398] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:40.483506 18472 tablet_bootstrap.cc:492] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:40.484066 18474 tablet_bootstrap.cc:654] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.485406 18472 tablet_bootstrap.cc:654] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:40.489434 18472 tablet_bootstrap.cc:492] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:40.489559 18474 tablet_bootstrap.cc:492] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:40.489605 18472 ts_tablet_manager.cc:1367] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:25:40.489755 18474 ts_tablet_manager.cc:1367] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:25:40.490399 18472 raft_consensus.cc:348] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.490617 18474 raft_consensus.cc:348] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.490653 18472 raft_consensus.cc:374] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.490868 18474 raft_consensus.cc:374] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:40.490898 18472 raft_consensus.cc:729] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.490965 18474 raft_consensus.cc:729] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:40.491220 18474 consensus_queue.cc:260] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.491230 18472 consensus_queue.cc:260] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.491744 18474 ts_tablet_manager.cc:1398] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:40.491813 18472 ts_tablet_manager.cc:1398] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:40.511618 18311 raft_consensus.cc:482] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:40.511822 18311 raft_consensus.cc:504] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.512585 18311 leader_election.cc:288] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:40.513196 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf03c730e341b8bc3edb9a72d03d39" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:40.513234 17196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf03c730e341b8bc3edb9a72d03d39" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:40.513474 17736 raft_consensus.cc:2454] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:40.513495 17196 raft_consensus.cc:2454] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:40.514014 17401 leader_election.cc:302] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, 8c66e3c3cad649daa465f3ae946f95f7; no voters:
I20240502 21:25:40.514366 18311 raft_consensus.cc:2781] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:40.514508 18311 raft_consensus.cc:482] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:40.514600 18311 raft_consensus.cc:3037] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.516707 18311 raft_consensus.cc:504] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.517457 18311 leader_election.cc:288] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:40.518047 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf03c730e341b8bc3edb9a72d03d39" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:40.518049 17196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf03c730e341b8bc3edb9a72d03d39" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:40.518294 17736 raft_consensus.cc:3037] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.518464 17196 raft_consensus.cc:3037] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.521507 17196 raft_consensus.cc:2454] T 8faf03c730e341b8bc3edb9a72d03d39 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:40.521525 17736 raft_consensus.cc:2454] T 8faf03c730e341b8bc3edb9a72d03d39 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:40.522056 17401 leader_election.cc:302] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:40.522462 18311 raft_consensus.cc:2781] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:40.522897 18311 raft_consensus.cc:686] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:40.523243 18311 consensus_queue.cc:237] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.526787 16915 catalog_manager.cc:5484] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:40.529181 18311 raft_consensus.cc:482] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:40.529423 18311 raft_consensus.cc:504] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.530179 18311 leader_election.cc:288] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:40.530802 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbccee0bdc9476c99e9926db224c5b9" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:40.530942 17196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbccee0bdc9476c99e9926db224c5b9" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:40.531100 17736 raft_consensus.cc:2454] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:40.531176 17196 raft_consensus.cc:2454] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:40.531636 17401 leader_election.cc:302] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:40.531949 18311 raft_consensus.cc:2781] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:40.532074 18311 raft_consensus.cc:482] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:40.532199 18311 raft_consensus.cc:3037] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.534198 18311 raft_consensus.cc:504] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.534899 18311 leader_election.cc:288] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:40.535395 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbccee0bdc9476c99e9926db224c5b9" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:40.535456 17196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cbccee0bdc9476c99e9926db224c5b9" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:40.535632 17196 raft_consensus.cc:3037] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.535655 17736 raft_consensus.cc:3037] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.538391 17196 raft_consensus.cc:2454] T 1cbccee0bdc9476c99e9926db224c5b9 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:40.538862 17401 leader_election.cc:302] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:40.538906 17736 raft_consensus.cc:2454] T 1cbccee0bdc9476c99e9926db224c5b9 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:40.539201 18311 raft_consensus.cc:2781] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:40.539426 18311 raft_consensus.cc:686] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:40.539726 18311 consensus_queue.cc:237] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:40.543169 16915 catalog_manager.cc:5484] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:40.806764 18435 raft_consensus.cc:482] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:40.806953 18435 raft_consensus.cc:504] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.807612 18435 leader_election.cc:288] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:40.808176 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5df765230c4f45bbd53d2e1e603aab" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:40.808226 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5df765230c4f45bbd53d2e1e603aab" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:40.808462 17333 raft_consensus.cc:2454] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:40.808463 17602 raft_consensus.cc:2454] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:40.808961 16995 leader_election.cc:302] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:40.809283 18435 raft_consensus.cc:2781] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:40.809474 18435 raft_consensus.cc:482] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:40.809583 18435 raft_consensus.cc:3037] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.811745 18435 raft_consensus.cc:504] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.812319 18435 leader_election.cc:288] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Requested vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:40.812973 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5df765230c4f45bbd53d2e1e603aab" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:40.813022 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5df765230c4f45bbd53d2e1e603aab" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:40.813263 17602 raft_consensus.cc:3037] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.813340 17333 raft_consensus.cc:3037] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:40.816323 17602 raft_consensus.cc:2454] T 1c5df765230c4f45bbd53d2e1e603aab P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:40.816330 17333 raft_consensus.cc:2454] T 1c5df765230c4f45bbd53d2e1e603aab P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:40.816840 16995 leader_election.cc:302] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:40.817257 18435 raft_consensus.cc:2781] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:40.817735 18435 raft_consensus.cc:686] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Running, Role: LEADER
I20240502 21:25:40.818070 18435 consensus_queue.cc:237] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:40.821276 16915 catalog_manager.cc:5484] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "5a2cdeb7351f4fffb44146f4d9365403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:40.979336 18487 consensus_queue.cc:1035] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:40.986534 18496 consensus_queue.cc:1035] T 1cbccee0bdc9476c99e9926db224c5b9 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:41.098728 18496 consensus_queue.cc:1035] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:41.102111 18496 consensus_queue.cc:1035] T 8faf03c730e341b8bc3edb9a72d03d39 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:41.276624 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:41.277065 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:41.286634 18493 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 21:25:41.377080 18435 consensus_queue.cc:1035] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:41.381278 18435 consensus_queue.cc:1035] T 1c5df765230c4f45bbd53d2e1e603aab P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 266.743 ms
time per row: 0.133372 ms
I20240502 21:25:41.719060 16913 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:41.729249 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.729537 17175 tablet_service.cc:1467] Processing CreateTablet for tablet 45d1c89db943440486f4f1efa57fcceb (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.729694 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.729979 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d1c89db943440486f4f1efa57fcceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.734795 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.734872 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.735311 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.735365 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.736167 18560 tablet_bootstrap.cc:492] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:41.738242 18560 tablet_bootstrap.cc:654] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.738520 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 45d1c89db943440486f4f1efa57fcceb (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.738693 17448 tablet_service.cc:1467] Processing CreateTablet for tablet 45d1c89db943440486f4f1efa57fcceb (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.739233 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d1c89db943440486f4f1efa57fcceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.739588 17313 tablet_service.cc:1467] Processing CreateTablet for tablet 27ff128fa4a245b59130a1c8150ed1c0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.739630 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 27ff128fa4a245b59130a1c8150ed1c0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.740001 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ff128fa4a245b59130a1c8150ed1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.740232 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ff128fa4a245b59130a1c8150ed1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.746661 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d1c89db943440486f4f1efa57fcceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.747881 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 27ff128fa4a245b59130a1c8150ed1c0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d40178b04e07467b932b29b3ef807732]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.748345 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ff128fa4a245b59130a1c8150ed1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.762037 18565 tablet_bootstrap.cc:492] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:41.767164 18562 tablet_bootstrap.cc:492] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:41.767778 18564 tablet_bootstrap.cc:492] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:41.769073 18566 tablet_bootstrap.cc:492] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:41.769718 18565 tablet_bootstrap.cc:654] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.770061 18564 tablet_bootstrap.cc:654] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.772217 18562 tablet_bootstrap.cc:654] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.772747 18563 tablet_bootstrap.cc:492] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:41.772883 18560 tablet_bootstrap.cc:492] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:41.773145 18560 ts_tablet_manager.cc:1367] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.014s
I20240502 21:25:41.774112 18566 tablet_bootstrap.cc:654] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.774204 18564 tablet_bootstrap.cc:492] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:41.774049 18560 raft_consensus.cc:348] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.774292 18560 raft_consensus.cc:374] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.774438 18560 raft_consensus.cc:729] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.774773 18560 consensus_queue.cc:260] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.774812 18563 tablet_bootstrap.cc:654] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.774945 18564 ts_tablet_manager.cc:1367] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:25:41.775977 18564 raft_consensus.cc:348] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.776621 18564 raft_consensus.cc:374] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.776803 18564 raft_consensus.cc:729] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.777240 18560 ts_tablet_manager.cc:1398] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:25:41.778292 18564 consensus_queue.cc:260] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.779201 18564 ts_tablet_manager.cc:1398] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:41.779621 18564 tablet_bootstrap.cc:492] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:41.779803 18565 tablet_bootstrap.cc:492] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:41.779999 18565 ts_tablet_manager.cc:1367] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20240502 21:25:41.781319 18564 tablet_bootstrap.cc:654] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.781234 18565 raft_consensus.cc:348] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.781508 18565 raft_consensus.cc:374] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.781612 18565 raft_consensus.cc:729] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.781874 18565 consensus_queue.cc:260] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.784092 18565 ts_tablet_manager.cc:1398] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:25:41.784492 18565 tablet_bootstrap.cc:492] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:41.785092 18562 tablet_bootstrap.cc:492] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:41.785375 18562 ts_tablet_manager.cc:1367] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240502 21:25:41.786319 18562 raft_consensus.cc:348] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.786695 18562 raft_consensus.cc:374] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.786859 18562 raft_consensus.cc:729] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.787348 18565 tablet_bootstrap.cc:654] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.787199 18562 consensus_queue.cc:260] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.787784 18562 ts_tablet_manager.cc:1398] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:41.791106 18564 tablet_bootstrap.cc:492] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:41.791368 18564 ts_tablet_manager.cc:1367] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240502 21:25:41.792404 18564 raft_consensus.cc:348] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.792739 18564 raft_consensus.cc:374] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.792872 18564 raft_consensus.cc:729] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.793263 18564 consensus_queue.cc:260] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.794407 18563 tablet_bootstrap.cc:492] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:41.794584 18563 ts_tablet_manager.cc:1367] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20240502 21:25:41.795554 18564 ts_tablet_manager.cc:1398] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:41.795445 18563 raft_consensus.cc:348] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.795661 18563 raft_consensus.cc:374] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.795675 18566 tablet_bootstrap.cc:492] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:41.795751 18563 raft_consensus.cc:729] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.795836 18566 ts_tablet_manager.cc:1367] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20240502 21:25:41.795990 18563 consensus_queue.cc:260] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.796627 18563 ts_tablet_manager.cc:1398] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:41.796669 18565 tablet_bootstrap.cc:492] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:41.796833 18565 ts_tablet_manager.cc:1367] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 21:25:41.796710 18566 raft_consensus.cc:348] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.797014 18566 raft_consensus.cc:374] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.797117 18566 raft_consensus.cc:729] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.797362 18566 consensus_queue.cc:260] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.798326 18563 tablet_bootstrap.cc:492] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:41.798964 18566 ts_tablet_manager.cc:1398] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:41.798847 18565 raft_consensus.cc:348] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.799082 18565 raft_consensus.cc:374] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.799163 18565 raft_consensus.cc:729] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.799461 18565 consensus_queue.cc:260] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.800027 18565 ts_tablet_manager.cc:1398] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:41.801160 18563 tablet_bootstrap.cc:654] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.804644 18563 tablet_bootstrap.cc:492] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:41.804878 18563 ts_tablet_manager.cc:1367] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 21:25:41.805691 18563 raft_consensus.cc:348] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.805868 18563 raft_consensus.cc:374] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.805971 18563 raft_consensus.cc:729] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.806222 18563 consensus_queue.cc:260] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.806799 18563 ts_tablet_manager.cc:1398] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:41.813647 18496 raft_consensus.cc:482] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:41.813831 18496 raft_consensus.cc:504] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.814646 18496 leader_election.cc:288] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:41.815212 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c89db943440486f4f1efa57fcceb" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:41.815236 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c89db943440486f4f1efa57fcceb" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:41.815507 17735 raft_consensus.cc:2454] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:41.815506 17197 raft_consensus.cc:2454] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:41.816022 17401 leader_election.cc:302] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, 8c66e3c3cad649daa465f3ae946f95f7; no voters:
I20240502 21:25:41.816334 18496 raft_consensus.cc:2781] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:41.816445 18496 raft_consensus.cc:482] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:41.816569 18496 raft_consensus.cc:3037] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.818784 18496 raft_consensus.cc:504] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.819526 18496 leader_election.cc:288] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:41.820003 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c89db943440486f4f1efa57fcceb" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:41.820084 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c89db943440486f4f1efa57fcceb" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:41.820289 17197 raft_consensus.cc:3037] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.820333 17735 raft_consensus.cc:3037] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.823395 17735 raft_consensus.cc:2454] T 45d1c89db943440486f4f1efa57fcceb P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:41.823395 17197 raft_consensus.cc:2454] T 45d1c89db943440486f4f1efa57fcceb P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:41.823940 17401 leader_election.cc:302] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:41.824265 18496 raft_consensus.cc:2781] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.824465 18496 raft_consensus.cc:686] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:41.824821 18496 consensus_queue.cc:237] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:41.828117 16915 catalog_manager.cc:5484] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:41.910851 18376 raft_consensus.cc:482] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:41.911041 18376 raft_consensus.cc:504] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.911842 18376 leader_election.cc:288] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:41.912447 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ff128fa4a245b59130a1c8150ed1c0" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:41.912467 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ff128fa4a245b59130a1c8150ed1c0" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:41.912693 17062 raft_consensus.cc:2454] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:41.912706 17602 raft_consensus.cc:2454] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 0.
I20240502 21:25:41.913132 17265 leader_election.cc:302] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:41.913432 18376 raft_consensus.cc:2781] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:41.913549 18376 raft_consensus.cc:482] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:41.913656 18376 raft_consensus.cc:3037] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.915648 18376 raft_consensus.cc:504] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.916177 18376 leader_election.cc:288] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Requested vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947)
I20240502 21:25:41.916702 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ff128fa4a245b59130a1c8150ed1c0" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:41.916743 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ff128fa4a245b59130a1c8150ed1c0" candidate_uuid: "967c7c03fe754179ab1544965af0cf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:41.916950 17062 raft_consensus.cc:3037] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.917008 17602 raft_consensus.cc:3037] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.919174 17062 raft_consensus.cc:2454] T 27ff128fa4a245b59130a1c8150ed1c0 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:41.919214 17602 raft_consensus.cc:2454] T 27ff128fa4a245b59130a1c8150ed1c0 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967c7c03fe754179ab1544965af0cf8d in term 1.
I20240502 21:25:41.919653 17265 leader_election.cc:302] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:41.919981 18376 raft_consensus.cc:2781] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.920449 18376 raft_consensus.cc:686] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [term 1 LEADER]: Becoming Leader. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Running, Role: LEADER
I20240502 21:25:41.920737 18376 consensus_queue.cc:237] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } }
I20240502 21:25:41.924077 16915 catalog_manager.cc:5484] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 967c7c03fe754179ab1544965af0cf8d (127.16.122.3). New cstate: current_term: 1 leader_uuid: "967c7c03fe754179ab1544965af0cf8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:41.952318 18455 raft_consensus.cc:482] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:41.952518 18455 raft_consensus.cc:504] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.953264 18455 leader_election.cc:288] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:41.953795 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3e5b8ed4fa4dbbb9d28453f7b55b7b" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:41.953946 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3e5b8ed4fa4dbbb9d28453f7b55b7b" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:41.954046 17062 raft_consensus.cc:2454] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:41.954211 17333 raft_consensus.cc:2454] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:41.954598 17534 leader_election.cc:302] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:41.954887 18455 raft_consensus.cc:2781] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:41.955019 18455 raft_consensus.cc:482] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:41.955144 18455 raft_consensus.cc:3037] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.957059 18455 raft_consensus.cc:504] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.957702 18455 leader_election.cc:288] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:41.958215 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3e5b8ed4fa4dbbb9d28453f7b55b7b" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:41.958305 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3e5b8ed4fa4dbbb9d28453f7b55b7b" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:41.958451 17333 raft_consensus.cc:3037] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.958577 17062 raft_consensus.cc:3037] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.960393 17333 raft_consensus.cc:2454] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:41.960448 17062 raft_consensus.cc:2454] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:41.960784 17534 leader_election.cc:302] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:41.961088 18455 raft_consensus.cc:2781] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.961529 18455 raft_consensus.cc:686] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:41.961874 18455 consensus_queue.cc:237] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:41.965173 16915 catalog_manager.cc:5484] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:42.253415 18496 consensus_queue.cc:1035] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.261157 18496 consensus_queue.cc:1035] T 45d1c89db943440486f4f1efa57fcceb P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:42.364605 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:42.364938 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:42.372993 18580 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:42.382905 18376 consensus_queue.cc:1035] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.386139 18376 consensus_queue.cc:1035] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 21:25:42.562319 18455 consensus_queue.cc:1035] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:42.564069 18455 consensus_queue.cc:1035] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:42.677491 17534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.3:39535, user_credentials={real_user=slave}} blocked reactor thread for 58055.2us
INSERT report
rows total: 2000
time total: 331.67 ms
time per row: 0.165835 ms
I20240502 21:25:42.887984 16912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:42.898334 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 5d668e9869144eb88bd21b74eef3a6a2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.898367 17448 tablet_service.cc:1467] Processing CreateTablet for tablet 5d668e9869144eb88bd21b74eef3a6a2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.898892 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d668e9869144eb88bd21b74eef3a6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.898892 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d668e9869144eb88bd21b74eef3a6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.898828 17175 tablet_service.cc:1467] Processing CreateTablet for tablet 5d668e9869144eb88bd21b74eef3a6a2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.899272 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d668e9869144eb88bd21b74eef3a6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.904271 17447 tablet_service.cc:1467] Processing CreateTablet for tablet 7886bd3d3e2048c8900a130f8f516082 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.904767 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7886bd3d3e2048c8900a130f8f516082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.904639 17313 tablet_service.cc:1467] Processing CreateTablet for tablet 8c15b912488947c0a3f29cc19e5fce61 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.905382 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c15b912488947c0a3f29cc19e5fce61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.906826 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 8c15b912488947c0a3f29cc19e5fce61 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.907260 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c15b912488947c0a3f29cc19e5fce61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.909999 17176 tablet_service.cc:1467] Processing CreateTablet for tablet 7886bd3d3e2048c8900a130f8f516082 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.910485 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7886bd3d3e2048c8900a130f8f516082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.914795 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 7886bd3d3e2048c8900a130f8f516082 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.915387 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7886bd3d3e2048c8900a130f8f516082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.917012 18651 tablet_bootstrap.cc:492] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:42.918159 18650 tablet_bootstrap.cc:492] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:42.919202 18651 tablet_bootstrap.cc:654] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.919881 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 8c15b912488947c0a3f29cc19e5fce61 (DEFAULT_TABLE table=rebalance_test_table_9 [id=d0b68de3e8a045b69746df07e711f0f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:42.920509 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c15b912488947c0a3f29cc19e5fce61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:42.921397 18650 tablet_bootstrap.cc:654] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.927523 18651 tablet_bootstrap.cc:492] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:42.928310 18651 ts_tablet_manager.cc:1367] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:25:42.929247 18651 raft_consensus.cc:348] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:42.929515 18651 raft_consensus.cc:374] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.929632 18651 raft_consensus.cc:729] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.929870 18651 consensus_queue.cc:260] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:42.934303 18651 ts_tablet_manager.cc:1398] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:25:42.936147 18651 tablet_bootstrap.cc:492] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:42.940644 18651 tablet_bootstrap.cc:654] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.942811 18652 tablet_bootstrap.cc:492] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:42.943703 18656 tablet_bootstrap.cc:492] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:42.944389 18655 tablet_bootstrap.cc:492] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:42.944828 18652 tablet_bootstrap.cc:654] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.945341 18650 tablet_bootstrap.cc:492] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:42.945535 18650 ts_tablet_manager.cc:1367] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240502 21:25:42.947193 18657 tablet_bootstrap.cc:492] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:42.949191 18655 tablet_bootstrap.cc:654] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.949189 18656 tablet_bootstrap.cc:654] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.950683 18650 raft_consensus.cc:348] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:42.953119 18657 tablet_bootstrap.cc:654] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.958037 18656 tablet_bootstrap.cc:492] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:42.958410 18656 ts_tablet_manager.cc:1367] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 21:25:42.959411 18650 raft_consensus.cc:374] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.959673 18650 raft_consensus.cc:729] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.959721 18656 raft_consensus.cc:348] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.960188 18656 raft_consensus.cc:374] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.960096 18650 consensus_queue.cc:260] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:42.960379 18656 raft_consensus.cc:729] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.960803 18656 consensus_queue.cc:260] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.961273 18650 ts_tablet_manager.cc:1398] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:25:42.961751 18650 tablet_bootstrap.cc:492] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:42.966233 18650 tablet_bootstrap.cc:654] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:42.968459 18652 tablet_bootstrap.cc:492] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:42.968624 18652 ts_tablet_manager.cc:1367] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.016s
I20240502 21:25:42.969156 18656 ts_tablet_manager.cc:1398] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:25:42.970484 18652 raft_consensus.cc:348] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.970721 18628 raft_consensus.cc:482] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:42.970945 18628 raft_consensus.cc:504] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.971604 18650 tablet_bootstrap.cc:492] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:42.971822 18652 raft_consensus.cc:374] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.971913 18652 raft_consensus.cc:729] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.972046 18628 leader_election.cc:288] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:42.972143 18652 consensus_queue.cc:260] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.973513 18650 ts_tablet_manager.cc:1367] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240502 21:25:42.974038 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15b912488947c0a3f29cc19e5fce61" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:42.974320 18650 raft_consensus.cc:348] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:42.974568 18650 raft_consensus.cc:374] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.974632 18650 raft_consensus.cc:729] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.972744 18652 ts_tablet_manager.cc:1398] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:25:42.974882 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15b912488947c0a3f29cc19e5fce61" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:42.975039 18650 consensus_queue.cc:260] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:42.975193 17333 raft_consensus.cc:2454] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:42.975597 18650 ts_tablet_manager.cc:1398] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20240502 21:25:42.976214 17534 leader_election.cc:341] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:42.977033 17534 leader_election.cc:302] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters: 5a2cdeb7351f4fffb44146f4d9365403
I20240502 21:25:42.977787 18628 raft_consensus.cc:2781] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:42.977926 18628 raft_consensus.cc:482] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:42.978039 18628 raft_consensus.cc:3037] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.982125 18628 raft_consensus.cc:504] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.985956 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15b912488947c0a3f29cc19e5fce61" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:42.985962 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15b912488947c0a3f29cc19e5fce61" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:42.986202 17333 raft_consensus.cc:3037] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:42.988307 17333 raft_consensus.cc:2454] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:42.989414 18651 tablet_bootstrap.cc:492] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:42.989677 18651 ts_tablet_manager.cc:1367] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240502 21:25:42.985345 18628 leader_election.cc:288] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
W20240502 21:25:42.986732 17534 leader_election.cc:341] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:42.990953 17534 leader_election.cc:302] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d; no voters: 5a2cdeb7351f4fffb44146f4d9365403
I20240502 21:25:42.991130 18651 raft_consensus.cc:348] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:42.991384 18651 raft_consensus.cc:374] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.991456 18651 raft_consensus.cc:729] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.991554 18628 raft_consensus.cc:2781] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:42.991729 18628 raft_consensus.cc:686] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:42.992024 18628 consensus_queue.cc:237] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.992192 18651 consensus_queue.cc:260] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:42.992581 18655 tablet_bootstrap.cc:492] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:42.992799 18655 ts_tablet_manager.cc:1367] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.049s user 0.028s sys 0.008s
I20240502 21:25:42.994113 18655 raft_consensus.cc:348] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:42.994493 18655 raft_consensus.cc:374] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.994645 18655 raft_consensus.cc:729] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.994989 18655 consensus_queue.cc:260] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:42.995407 18651 ts_tablet_manager.cc:1398] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20240502 21:25:42.996935 18657 tablet_bootstrap.cc:492] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:42.997128 18657 ts_tablet_manager.cc:1367] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240502 21:25:42.997426 16916 catalog_manager.cc:5484] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:42.998075 18657 raft_consensus.cc:348] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:42.998318 18657 raft_consensus.cc:374] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:42.998545 18657 raft_consensus.cc:729] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:42.998823 18657 consensus_queue.cc:260] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:43.001852 18657 ts_tablet_manager.cc:1398] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:25:43.002768 18655 ts_tablet_manager.cc:1398] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 21:25:43.003172 18655 tablet_bootstrap.cc:492] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:43.004840 18655 tablet_bootstrap.cc:654] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.035006 18655 tablet_bootstrap.cc:492] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:43.035295 18655 ts_tablet_manager.cc:1367] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240502 21:25:43.036036 18655 raft_consensus.cc:348] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:43.036339 18655 raft_consensus.cc:374] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.036475 18655 raft_consensus.cc:729] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.036840 18655 consensus_queue.cc:260] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:43.037385 18655 ts_tablet_manager.cc:1398] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:43.090924 18583 raft_consensus.cc:482] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.091179 18583 raft_consensus.cc:504] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:43.092038 18583 leader_election.cc:288] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:43.092590 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d668e9869144eb88bd21b74eef3a6a2" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:43.092677 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d668e9869144eb88bd21b74eef3a6a2" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:43.092823 17735 raft_consensus.cc:2454] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:43.092934 17197 raft_consensus.cc:2454] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 0.
I20240502 21:25:43.093381 17401 leader_election.cc:302] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, 8c66e3c3cad649daa465f3ae946f95f7; no voters:
I20240502 21:25:43.093640 18583 raft_consensus.cc:2781] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:43.093763 18583 raft_consensus.cc:482] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:43.093861 18583 raft_consensus.cc:3037] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.096324 18583 raft_consensus.cc:504] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:43.096941 18583 leader_election.cc:288] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Requested vote from peers 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:43.098915 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d668e9869144eb88bd21b74eef3a6a2" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:43.098918 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d668e9869144eb88bd21b74eef3a6a2" candidate_uuid: "8c66e3c3cad649daa465f3ae946f95f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:43.099164 17197 raft_consensus.cc:3037] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.099195 17735 raft_consensus.cc:3037] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.111979 18490 raft_consensus.cc:482] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.112234 18490 raft_consensus.cc:504] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:43.113229 18490 leader_election.cc:288] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:43.113896 17196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886bd3d3e2048c8900a130f8f516082" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:43.114180 17196 raft_consensus.cc:2454] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:43.114403 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886bd3d3e2048c8900a130f8f516082" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:43.114646 17668 leader_election.cc:302] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:43.114697 17468 raft_consensus.cc:2454] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:43.114957 18490 raft_consensus.cc:2781] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:43.115166 18490 raft_consensus.cc:482] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:43.115271 18490 raft_consensus.cc:3037] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.139289 17735 raft_consensus.cc:2454] T 5d668e9869144eb88bd21b74eef3a6a2 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:43.139289 17197 raft_consensus.cc:2454] T 5d668e9869144eb88bd21b74eef3a6a2 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c66e3c3cad649daa465f3ae946f95f7 in term 1.
I20240502 21:25:43.140022 17401 leader_election.cc:302] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:43.140462 18583 raft_consensus.cc:2781] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.140702 18583 raft_consensus.cc:686] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Running, Role: LEADER
I20240502 21:25:43.141113 18583 consensus_queue.cc:237] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:43.142477 18490 raft_consensus.cc:504] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:43.143477 18490 leader_election.cc:288] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Requested vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287)
I20240502 21:25:43.143870 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886bd3d3e2048c8900a130f8f516082" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:43.144109 17468 raft_consensus.cc:3037] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.144212 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886bd3d3e2048c8900a130f8f516082" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:43.144531 17197 raft_consensus.cc:3037] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.145119 16916 catalog_manager.cc:5484] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "8c66e3c3cad649daa465f3ae946f95f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:43.211360 17468 raft_consensus.cc:2454] T 7886bd3d3e2048c8900a130f8f516082 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:43.211354 17197 raft_consensus.cc:2454] T 7886bd3d3e2048c8900a130f8f516082 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:43.212021 17668 leader_election.cc:302] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:43.212401 18490 raft_consensus.cc:2781] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.212848 18490 raft_consensus.cc:686] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 LEADER]: Becoming Leader. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Running, Role: LEADER
I20240502 21:25:43.213225 18490 consensus_queue.cc:237] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:43.216727 16913 catalog_manager.cc:5484] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6). New cstate: current_term: 1 leader_uuid: "7181173c030d4a6db6e29c158fdb9a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:43.298754 18542 raft_consensus.cc:482] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.298979 18542 raft_consensus.cc:504] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:43.300343 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15b912488947c0a3f29cc19e5fce61" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:43.300340 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15b912488947c0a3f29cc19e5fce61" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:43.300690 17333 raft_consensus.cc:2379] T 8c15b912488947c0a3f29cc19e5fce61 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a2cdeb7351f4fffb44146f4d9365403 in current term 1: Already voted for candidate 86a457a4beb4478b8a34e3633f8d2238 in this term.
I20240502 21:25:43.300976 18542 leader_election.cc:288] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:43.301211 16995 leader_election.cc:302] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403; no voters: 86a457a4beb4478b8a34e3633f8d2238, 967c7c03fe754179ab1544965af0cf8d
I20240502 21:25:43.301506 18542 raft_consensus.cc:3037] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.304967 18542 raft_consensus.cc:2726] T 8c15b912488947c0a3f29cc19e5fce61 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:43.444795 18628 consensus_queue.cc:1035] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:43.451522 18629 consensus_queue.cc:1035] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:43.643445 18583 consensus_queue.cc:1035] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:43.658254 18680 consensus_queue.cc:1035] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:43.678369 18669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:43.678730 18669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:43.686683 18669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 21:25:43.750389 18490 consensus_queue.cc:1035] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:43.784070 18490 consensus_queue.cc:1035] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 21:25:43.944582 17131 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:36081, user_credentials={real_user=slave}} blocked reactor thread for 54121.7us
W20240502 21:25:44.062871 17670 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:36081, user_credentials={real_user=slave}} blocked reactor thread for 110132us
INSERT report
rows total: 2000
time total: 347.521 ms
time per row: 0.173761 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:45.167827 16916 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:45.177839 17042 tablet_service.cc:1467] Processing CreateTablet for tablet 4a18b7abc37b41bbb841dc9af174e035 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.178412 17042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a18b7abc37b41bbb841dc9af174e035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.178566 17313 tablet_service.cc:1467] Processing CreateTablet for tablet 4a18b7abc37b41bbb841dc9af174e035 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.178617 17447 tablet_service.cc:1467] Processing CreateTablet for tablet 8637ea61f53c47298f98d7e2b3f7d34f (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.178997 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a18b7abc37b41bbb841dc9af174e035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.179064 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8637ea61f53c47298f98d7e2b3f7d34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.182842 17176 tablet_service.cc:1467] Processing CreateTablet for tablet 8637ea61f53c47298f98d7e2b3f7d34f (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.183007 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 4a18b7abc37b41bbb841dc9af174e035 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.183302 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8637ea61f53c47298f98d7e2b3f7d34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.183450 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a18b7abc37b41bbb841dc9af174e035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.184322 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 8637ea61f53c47298f98d7e2b3f7d34f (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.184729 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8637ea61f53c47298f98d7e2b3f7d34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.190920 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 8d59bf301e264f25b94f4bb93d4bfd82 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.191474 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d59bf301e264f25b94f4bb93d4bfd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.195906 17041 tablet_service.cc:1467] Processing CreateTablet for tablet 8d59bf301e264f25b94f4bb93d4bfd82 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.195983 17312 tablet_service.cc:1467] Processing CreateTablet for tablet 8d59bf301e264f25b94f4bb93d4bfd82 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d0ecec4ae2e7448a892f8be855768ff6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.196272 18754 tablet_bootstrap.cc:492] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:45.196445 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d59bf301e264f25b94f4bb93d4bfd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.196476 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d59bf301e264f25b94f4bb93d4bfd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.202294 18752 tablet_bootstrap.cc:492] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:45.203173 18751 tablet_bootstrap.cc:492] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:45.204699 18752 tablet_bootstrap.cc:654] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.208434 18751 tablet_bootstrap.cc:654] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.208608 18754 tablet_bootstrap.cc:654] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.214054 18752 tablet_bootstrap.cc:492] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:45.214324 18752 ts_tablet_manager.cc:1367] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 21:25:45.215564 18752 raft_consensus.cc:348] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.215920 18752 raft_consensus.cc:374] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.216089 18752 raft_consensus.cc:729] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.216451 18752 consensus_queue.cc:260] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.217434 18752 ts_tablet_manager.cc:1398] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:45.218895 18757 tablet_bootstrap.cc:492] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:45.221307 18757 tablet_bootstrap.cc:654] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.223076 18753 tablet_bootstrap.cc:492] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:45.225302 18754 tablet_bootstrap.cc:492] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:45.225782 18753 tablet_bootstrap.cc:654] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.225540 18754 ts_tablet_manager.cc:1367] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 21:25:45.227527 18750 tablet_bootstrap.cc:492] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:45.228156 18757 tablet_bootstrap.cc:492] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:45.228029 18754 raft_consensus.cc:348] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.228328 18757 ts_tablet_manager.cc:1367] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:25:45.228343 18754 raft_consensus.cc:374] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.228508 18754 raft_consensus.cc:729] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.228896 18750 tablet_bootstrap.cc:654] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.228793 18754 consensus_queue.cc:260] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.229193 18757 raft_consensus.cc:348] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.229411 18757 raft_consensus.cc:374] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.229496 18754 ts_tablet_manager.cc:1398] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:45.229507 18757 raft_consensus.cc:729] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.229776 18754 tablet_bootstrap.cc:492] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:45.229734 18757 consensus_queue.cc:260] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.230324 18757 ts_tablet_manager.cc:1398] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:45.231480 18753 tablet_bootstrap.cc:492] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:45.231626 18753 ts_tablet_manager.cc:1367] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240502 21:25:45.232424 18753 raft_consensus.cc:348] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.232635 18753 raft_consensus.cc:374] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.232726 18753 raft_consensus.cc:729] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.232936 18753 consensus_queue.cc:260] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.233538 18753 ts_tablet_manager.cc:1398] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:45.233943 18753 tablet_bootstrap.cc:492] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:45.235621 18753 tablet_bootstrap.cc:654] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.237573 18750 tablet_bootstrap.cc:492] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:45.237810 18754 tablet_bootstrap.cc:654] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.237859 18750 ts_tablet_manager.cc:1367] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:25:45.238987 18750 raft_consensus.cc:348] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.239324 18750 raft_consensus.cc:374] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.239485 18750 raft_consensus.cc:729] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.239790 18750 consensus_queue.cc:260] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.240619 18750 ts_tablet_manager.cc:1398] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:45.242278 18753 tablet_bootstrap.cc:492] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:45.242498 18753 ts_tablet_manager.cc:1367] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:25:45.243439 18753 raft_consensus.cc:348] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.243839 18753 raft_consensus.cc:374] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.243973 18753 raft_consensus.cc:729] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.244338 18753 consensus_queue.cc:260] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.244493 18751 tablet_bootstrap.cc:492] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:45.244942 18754 tablet_bootstrap.cc:492] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:45.245132 18751 ts_tablet_manager.cc:1367] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240502 21:25:45.245163 18754 ts_tablet_manager.cc:1367] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 21:25:45.245994 18754 raft_consensus.cc:348] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.246198 18753 ts_tablet_manager.cc:1398] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:45.246253 18754 raft_consensus.cc:374] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.246341 18754 raft_consensus.cc:729] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.246809 18754 consensus_queue.cc:260] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.246891 18751 raft_consensus.cc:348] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.247085 18751 raft_consensus.cc:374] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.247181 18751 raft_consensus.cc:729] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.247490 18751 consensus_queue.cc:260] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.247663 18754 ts_tablet_manager.cc:1398] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:45.249765 18751 ts_tablet_manager.cc:1398] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:25:45.250075 18751 tablet_bootstrap.cc:492] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:45.251909 18751 tablet_bootstrap.cc:654] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.253836 18751 tablet_bootstrap.cc:492] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:45.253989 18751 ts_tablet_manager.cc:1367] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:25:45.254639 18751 raft_consensus.cc:348] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.254896 18751 raft_consensus.cc:374] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.254971 18751 raft_consensus.cc:729] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.255247 18751 consensus_queue.cc:260] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.255894 18751 ts_tablet_manager.cc:1398] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:45.275259 18731 raft_consensus.cc:482] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:45.275449 18731 raft_consensus.cc:504] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.276113 18731 leader_election.cc:288] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:45.276628 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8637ea61f53c47298f98d7e2b3f7d34f" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" is_pre_election: true
I20240502 21:25:45.276693 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8637ea61f53c47298f98d7e2b3f7d34f" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:45.276844 17197 raft_consensus.cc:2454] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:45.276978 17468 raft_consensus.cc:2454] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 0.
I20240502 21:25:45.277310 17668 leader_election.cc:302] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:45.277572 18731 raft_consensus.cc:2781] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:45.277688 18731 raft_consensus.cc:482] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:45.277777 18731 raft_consensus.cc:3037] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.279726 18731 raft_consensus.cc:504] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.280280 18731 leader_election.cc:288] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Requested vote from peers ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287), 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081)
I20240502 21:25:45.280798 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8637ea61f53c47298f98d7e2b3f7d34f" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:45.281003 17468 raft_consensus.cc:3037] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.280963 17197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8637ea61f53c47298f98d7e2b3f7d34f" candidate_uuid: "7181173c030d4a6db6e29c158fdb9a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4ceec4b9204f9987cc97cee5d1d820"
I20240502 21:25:45.281136 17197 raft_consensus.cc:3037] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.281838 18745 raft_consensus.cc:482] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:45.282001 18745 raft_consensus.cc:504] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.283010 18745 leader_election.cc:288] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:45.283289 17468 raft_consensus.cc:2454] T 8637ea61f53c47298f98d7e2b3f7d34f P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:45.283718 17197 raft_consensus.cc:2454] T 8637ea61f53c47298f98d7e2b3f7d34f P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7181173c030d4a6db6e29c158fdb9a69 in term 1.
I20240502 21:25:45.284037 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d59bf301e264f25b94f4bb93d4bfd82" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403" is_pre_election: true
I20240502 21:25:45.284266 17061 raft_consensus.cc:2454] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:45.284577 17670 leader_election.cc:302] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, 8c66e3c3cad649daa465f3ae946f95f7; no voters:
I20240502 21:25:45.285169 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d59bf301e264f25b94f4bb93d4bfd82" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:45.285244 17534 leader_election.cc:302] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:45.285329 18731 raft_consensus.cc:2781] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.285435 17333 raft_consensus.cc:2454] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 0.
I20240502 21:25:45.285811 18731 raft_consensus.cc:686] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [term 1 LEADER]: Becoming Leader. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Running, Role: LEADER
I20240502 21:25:45.285971 18745 raft_consensus.cc:2781] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:45.286082 18745 raft_consensus.cc:482] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:45.286167 18745 raft_consensus.cc:3037] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.286178 18731 consensus_queue.cc:237] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:45.288519 18745 raft_consensus.cc:504] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.289074 18745 leader_election.cc:288] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Requested vote from peers 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:45.290088 16910 catalog_manager.cc:5484] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6). New cstate: current_term: 1 leader_uuid: "7181173c030d4a6db6e29c158fdb9a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:45.291951 17062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d59bf301e264f25b94f4bb93d4bfd82" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2cdeb7351f4fffb44146f4d9365403"
I20240502 21:25:45.292166 17062 raft_consensus.cc:3037] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.293330 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d59bf301e264f25b94f4bb93d4bfd82" candidate_uuid: "86a457a4beb4478b8a34e3633f8d2238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:45.293610 17333 raft_consensus.cc:3037] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.294721 17062 raft_consensus.cc:2454] T 8d59bf301e264f25b94f4bb93d4bfd82 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:45.295482 17534 leader_election.cc:302] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:45.296828 18768 raft_consensus.cc:2781] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.297392 18768 raft_consensus.cc:686] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 LEADER]: Becoming Leader. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Running, Role: LEADER
I20240502 21:25:45.298079 18768 consensus_queue.cc:237] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:45.298938 17333 raft_consensus.cc:2454] T 8d59bf301e264f25b94f4bb93d4bfd82 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a457a4beb4478b8a34e3633f8d2238 in term 1.
I20240502 21:25:45.303793 16910 catalog_manager.cc:5484] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5). New cstate: current_term: 1 leader_uuid: "86a457a4beb4478b8a34e3633f8d2238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:45.647866 18542 raft_consensus.cc:482] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:45.648075 18542 raft_consensus.cc:504] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.648689 18542 leader_election.cc:288] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:45.649420 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a18b7abc37b41bbb841dc9af174e035" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:45.649401 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a18b7abc37b41bbb841dc9af174e035" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:45.649675 17333 raft_consensus.cc:2454] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:45.649777 17602 raft_consensus.cc:2454] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:45.650282 16995 leader_election.cc:302] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:45.650681 18542 raft_consensus.cc:2781] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:45.650784 18542 raft_consensus.cc:482] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:45.650897 18542 raft_consensus.cc:3037] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.652778 18542 raft_consensus.cc:504] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.653416 18542 leader_election.cc:288] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Requested vote from peers 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211), 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535)
I20240502 21:25:45.653879 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a18b7abc37b41bbb841dc9af174e035" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:45.653905 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a18b7abc37b41bbb841dc9af174e035" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:45.654103 17333 raft_consensus.cc:3037] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.654124 17602 raft_consensus.cc:3037] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.657023 17602 raft_consensus.cc:2454] T 4a18b7abc37b41bbb841dc9af174e035 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:45.657038 17333 raft_consensus.cc:2454] T 4a18b7abc37b41bbb841dc9af174e035 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:45.657418 16995 leader_election.cc:302] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:45.657704 18542 raft_consensus.cc:2781] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.658176 18542 raft_consensus.cc:686] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Running, Role: LEADER
I20240502 21:25:45.658726 18542 consensus_queue.cc:237] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } }
I20240502 21:25:45.662174 16910 catalog_manager.cc:5484] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "5a2cdeb7351f4fffb44146f4d9365403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:45.796170 18731 consensus_queue.cc:1035] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:45.803432 18731 consensus_queue.cc:1035] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:45.898372 18768 consensus_queue.cc:1035] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:45.904770 18768 consensus_queue.cc:1035] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.154485 18542 consensus_queue.cc:1035] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:46.157967 18774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:46.158532 18774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:25:46.161281 18542 consensus_queue.cc:1035] T 4a18b7abc37b41bbb841dc9af174e035 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:46.169430 18774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
INSERT report
rows total: 2000
time total: 269.36 ms
time per row: 0.13468 ms
I20240502 21:25:46.620272 16918 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49182:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:46.631908 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 9edf4032df9c47caa29d56ef153d85cc (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.631914 17176 tablet_service.cc:1467] Processing CreateTablet for tablet 9edf4032df9c47caa29d56ef153d85cc (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.632295 17175 tablet_service.cc:1467] Processing CreateTablet for tablet 34b0d71e9d72459aaa0917655434cf47 (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.632431 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edf4032df9c47caa29d56ef153d85cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.632431 17176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edf4032df9c47caa29d56ef153d85cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.632719 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b0d71e9d72459aaa0917655434cf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.638578 18848 tablet_bootstrap.cc:492] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:46.638864 17447 tablet_service.cc:1467] Processing CreateTablet for tablet 9edf4032df9c47caa29d56ef153d85cc (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.639339 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edf4032df9c47caa29d56ef153d85cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.639395 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 34b0d71e9d72459aaa0917655434cf47 (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.639853 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b0d71e9d72459aaa0917655434cf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.640502 18848 tablet_bootstrap.cc:654] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.642812 18848 tablet_bootstrap.cc:492] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:46.642967 18848 ts_tablet_manager.cc:1367] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 21:25:46.644783 18848 raft_consensus.cc:348] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.645224 18848 raft_consensus.cc:374] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.645469 17448 tablet_service.cc:1467] Processing CreateTablet for tablet 34b0d71e9d72459aaa0917655434cf47 (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.645932 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b0d71e9d72459aaa0917655434cf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.646627 17041 tablet_service.cc:1467] Processing CreateTablet for tablet ff283b6ba42f4d75b0092ddfc23af7a2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.646274 18848 raft_consensus.cc:729] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.647119 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff283b6ba42f4d75b0092ddfc23af7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.647176 18848 consensus_queue.cc:260] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.647989 18848 ts_tablet_manager.cc:1398] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:46.648378 18848 tablet_bootstrap.cc:492] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820: Bootstrap starting.
I20240502 21:25:46.650477 18848 tablet_bootstrap.cc:654] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.650863 17580 tablet_service.cc:1467] Processing CreateTablet for tablet ff283b6ba42f4d75b0092ddfc23af7a2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.651082 17312 tablet_service.cc:1467] Processing CreateTablet for tablet ff283b6ba42f4d75b0092ddfc23af7a2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=99ba7c876cef419a9d0b9a45bd91d332]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.651468 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff283b6ba42f4d75b0092ddfc23af7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.651588 17312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff283b6ba42f4d75b0092ddfc23af7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.658003 18853 tablet_bootstrap.cc:492] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:46.660773 18850 tablet_bootstrap.cc:492] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:46.662746 18853 tablet_bootstrap.cc:654] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.665839 18848 tablet_bootstrap.cc:492] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820: No bootstrap required, opened a new log
I20240502 21:25:46.666260 18848 ts_tablet_manager.cc:1367] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.008s
I20240502 21:25:46.666339 18850 tablet_bootstrap.cc:654] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.667353 18848 raft_consensus.cc:348] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.667301 18854 tablet_bootstrap.cc:492] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238: Bootstrap starting.
I20240502 21:25:46.668363 18848 raft_consensus.cc:374] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.668566 18848 raft_consensus.cc:729] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.669087 18848 consensus_queue.cc:260] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.670212 18854 tablet_bootstrap.cc:654] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.670845 18852 tablet_bootstrap.cc:492] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403: Bootstrap starting.
I20240502 21:25:46.671270 18855 tablet_bootstrap.cc:492] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d: Bootstrap starting.
I20240502 21:25:46.672542 18852 tablet_bootstrap.cc:654] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.673566 18855 tablet_bootstrap.cc:654] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.675806 18848 ts_tablet_manager.cc:1398] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 21:25:46.682211 18850 tablet_bootstrap.cc:492] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:46.682590 18850 ts_tablet_manager.cc:1367] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.006s
I20240502 21:25:46.682782 18853 tablet_bootstrap.cc:492] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:46.683017 18854 tablet_bootstrap.cc:492] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238: No bootstrap required, opened a new log
I20240502 21:25:46.683015 18853 ts_tablet_manager.cc:1367] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240502 21:25:46.683279 18854 ts_tablet_manager.cc:1367] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.008s
I20240502 21:25:46.683616 18850 raft_consensus.cc:348] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.683959 18850 raft_consensus.cc:374] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.684032 18852 tablet_bootstrap.cc:492] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403: No bootstrap required, opened a new log
I20240502 21:25:46.684031 18853 raft_consensus.cc:348] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.684209 18852 ts_tablet_manager.cc:1367] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240502 21:25:46.684193 18850 raft_consensus.cc:729] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.684360 18853 raft_consensus.cc:374] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.684573 18853 raft_consensus.cc:729] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.684527 18850 consensus_queue.cc:260] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.685154 18853 consensus_queue.cc:260] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.685322 18850 ts_tablet_manager.cc:1398] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:46.685246 18852 raft_consensus.cc:348] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.685509 18852 raft_consensus.cc:374] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.685616 18852 raft_consensus.cc:729] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.685892 18852 consensus_queue.cc:260] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.686172 18853 ts_tablet_manager.cc:1398] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:46.686641 18852 ts_tablet_manager.cc:1398] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:46.684887 18854 raft_consensus.cc:348] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.687322 18853 tablet_bootstrap.cc:492] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69: Bootstrap starting.
I20240502 21:25:46.688695 18670 raft_consensus.cc:482] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:46.688926 18670 raft_consensus.cc:504] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.689116 18853 tablet_bootstrap.cc:654] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.690112 18670 leader_election.cc:288] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:46.690618 18855 tablet_bootstrap.cc:492] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d: No bootstrap required, opened a new log
I20240502 21:25:46.690752 18850 tablet_bootstrap.cc:492] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7: Bootstrap starting.
I20240502 21:25:46.690785 18855 ts_tablet_manager.cc:1367] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240502 21:25:46.691615 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edf4032df9c47caa29d56ef153d85cc" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:46.691653 18855 raft_consensus.cc:348] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.691872 18855 raft_consensus.cc:374] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.691876 17468 raft_consensus.cc:2454] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:46.691953 18855 raft_consensus.cc:729] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967c7c03fe754179ab1544965af0cf8d, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.692253 18855 consensus_queue.cc:260] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.692606 18850 tablet_bootstrap.cc:654] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.692904 17131 leader_election.cc:302] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:46.693234 18670 raft_consensus.cc:2781] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:46.693346 18670 raft_consensus.cc:482] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:46.693434 18670 raft_consensus.cc:3037] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.694432 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edf4032df9c47caa29d56ef153d85cc" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:46.695076 18855 ts_tablet_manager.cc:1398] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20240502 21:25:46.695513 17129 leader_election.cc:341] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:46.696136 18854 raft_consensus.cc:374] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.696379 18854 raft_consensus.cc:729] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a457a4beb4478b8a34e3633f8d2238, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.696828 18854 consensus_queue.cc:260] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.701256 18670 raft_consensus.cc:504] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.702226 18670 leader_election.cc:288] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:46.703236 18853 tablet_bootstrap.cc:492] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69: No bootstrap required, opened a new log
I20240502 21:25:46.703500 18853 ts_tablet_manager.cc:1367] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:25:46.704118 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edf4032df9c47caa29d56ef153d85cc" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:46.704391 17468 raft_consensus.cc:3037] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.704543 18853 raft_consensus.cc:348] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.704852 18853 raft_consensus.cc:374] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.704936 18853 raft_consensus.cc:729] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181173c030d4a6db6e29c158fdb9a69, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.705271 18853 consensus_queue.cc:260] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.707676 18853 ts_tablet_manager.cc:1398] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:46.708912 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edf4032df9c47caa29d56ef153d85cc" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:46.709136 18854 ts_tablet_manager.cc:1398] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240502 21:25:46.709156 17736 raft_consensus.cc:3037] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.709659 17468 raft_consensus.cc:2454] T 9edf4032df9c47caa29d56ef153d85cc P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:46.710045 18850 tablet_bootstrap.cc:492] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7: No bootstrap required, opened a new log
I20240502 21:25:46.710223 18850 ts_tablet_manager.cc:1367] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.012s
I20240502 21:25:46.711051 18850 raft_consensus.cc:348] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.711349 18850 raft_consensus.cc:374] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.711373 17131 leader_election.cc:302] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:46.711525 18850 raft_consensus.cc:729] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c66e3c3cad649daa465f3ae946f95f7, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.712041 17736 raft_consensus.cc:2454] T 9edf4032df9c47caa29d56ef153d85cc P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:46.712142 18670 raft_consensus.cc:2781] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.712005 18850 consensus_queue.cc:260] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.712292 18670 raft_consensus.cc:686] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:46.712651 18670 consensus_queue.cc:237] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } }
I20240502 21:25:46.713920 18850 ts_tablet_manager.cc:1398] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:25:46.716964 16910 catalog_manager.cc:5484] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:46.820188 18670 raft_consensus.cc:482] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:46.820421 18670 raft_consensus.cc:504] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.821039 18670 leader_election.cc:288] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:46.821429 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b0d71e9d72459aaa0917655434cf47" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7" is_pre_election: true
I20240502 21:25:46.821667 17468 raft_consensus.cc:2454] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:46.821713 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b0d71e9d72459aaa0917655434cf47" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69" is_pre_election: true
I20240502 21:25:46.821975 17736 raft_consensus.cc:2454] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 0.
I20240502 21:25:46.822057 17131 leader_election.cc:302] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c66e3c3cad649daa465f3ae946f95f7, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:46.822376 18670 raft_consensus.cc:2781] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:46.822487 18670 raft_consensus.cc:482] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:46.822597 18670 raft_consensus.cc:3037] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.824569 18670 raft_consensus.cc:504] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.825117 18670 leader_election.cc:288] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Requested vote from peers 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081), 7181173c030d4a6db6e29c158fdb9a69 (127.16.122.6:35119)
I20240502 21:25:46.825543 17468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b0d71e9d72459aaa0917655434cf47" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c66e3c3cad649daa465f3ae946f95f7"
I20240502 21:25:46.825585 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b0d71e9d72459aaa0917655434cf47" candidate_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7181173c030d4a6db6e29c158fdb9a69"
I20240502 21:25:46.825747 17468 raft_consensus.cc:3037] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.825758 17736 raft_consensus.cc:3037] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.827706 17468 raft_consensus.cc:2454] T 34b0d71e9d72459aaa0917655434cf47 P 8c66e3c3cad649daa465f3ae946f95f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:46.827718 17736 raft_consensus.cc:2454] T 34b0d71e9d72459aaa0917655434cf47 P 7181173c030d4a6db6e29c158fdb9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4ceec4b9204f9987cc97cee5d1d820 in term 1.
I20240502 21:25:46.828106 17129 leader_election.cc:302] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7181173c030d4a6db6e29c158fdb9a69, ae4ceec4b9204f9987cc97cee5d1d820; no voters:
I20240502 21:25:46.828433 18670 raft_consensus.cc:2781] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.828702 18670 raft_consensus.cc:686] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [term 1 LEADER]: Becoming Leader. State: Replica: ae4ceec4b9204f9987cc97cee5d1d820, State: Running, Role: LEADER
I20240502 21:25:46.828972 18670 consensus_queue.cc:237] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } }
I20240502 21:25:46.832716 16910 catalog_manager.cc:5484] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4ceec4b9204f9987cc97cee5d1d820" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:46.972318 18834 raft_consensus.cc:482] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:46.972518 18834 raft_consensus.cc:504] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.973266 18834 leader_election.cc:288] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:46.973639 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff283b6ba42f4d75b0092ddfc23af7a2" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d" is_pre_election: true
I20240502 21:25:46.973883 17333 raft_consensus.cc:2454] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:46.973984 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff283b6ba42f4d75b0092ddfc23af7a2" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238" is_pre_election: true
I20240502 21:25:46.974285 17602 raft_consensus.cc:2454] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 0.
I20240502 21:25:46.974387 16995 leader_election.cc:302] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 967c7c03fe754179ab1544965af0cf8d; no voters:
I20240502 21:25:46.974752 18834 raft_consensus.cc:2781] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:46.974895 18834 raft_consensus.cc:482] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:46.975001 18834 raft_consensus.cc:3037] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.976938 18834 raft_consensus.cc:504] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.977486 18834 leader_election.cc:288] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Requested vote from peers 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535), 86a457a4beb4478b8a34e3633f8d2238 (127.16.122.5:36211)
I20240502 21:25:46.978056 17602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff283b6ba42f4d75b0092ddfc23af7a2" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a457a4beb4478b8a34e3633f8d2238"
I20240502 21:25:46.978246 17602 raft_consensus.cc:3037] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.980190 17602 raft_consensus.cc:2454] T ff283b6ba42f4d75b0092ddfc23af7a2 P 86a457a4beb4478b8a34e3633f8d2238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:46.980545 17333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff283b6ba42f4d75b0092ddfc23af7a2" candidate_uuid: "5a2cdeb7351f4fffb44146f4d9365403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967c7c03fe754179ab1544965af0cf8d"
I20240502 21:25:46.980620 16996 leader_election.cc:302] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cdeb7351f4fffb44146f4d9365403, 86a457a4beb4478b8a34e3633f8d2238; no voters:
I20240502 21:25:46.980782 17333 raft_consensus.cc:3037] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.980937 18834 raft_consensus.cc:2781] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.981341 18834 raft_consensus.cc:686] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cdeb7351f4fffb44146f4d9365403, State: Running, Role: LEADER
I20240502 21:25:46.981650 18834 consensus_queue.cc:237] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } }
I20240502 21:25:46.984230 17333 raft_consensus.cc:2454] T ff283b6ba42f4d75b0092ddfc23af7a2 P 967c7c03fe754179ab1544965af0cf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cdeb7351f4fffb44146f4d9365403 in term 1.
I20240502 21:25:46.985606 16910 catalog_manager.cc:5484] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "5a2cdeb7351f4fffb44146f4d9365403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2cdeb7351f4fffb44146f4d9365403" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:47.194763 18670 consensus_queue.cc:1035] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:47.203507 18670 consensus_queue.cc:1035] T 9edf4032df9c47caa29d56ef153d85cc P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:47.293704 18875 consensus_queue.cc:1035] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c66e3c3cad649daa465f3ae946f95f7" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:47.300483 18670 consensus_queue.cc:1035] T 34b0d71e9d72459aaa0917655434cf47 P ae4ceec4b9204f9987cc97cee5d1d820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7181173c030d4a6db6e29c158fdb9a69" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:47.407335 18869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:47.407660 18869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:47.416153 18869 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:47.425199 18834 consensus_queue.cc:1035] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967c7c03fe754179ab1544965af0cf8d" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:47.433655 18867 consensus_queue.cc:1035] T ff283b6ba42f4d75b0092ddfc23af7a2 P 5a2cdeb7351f4fffb44146f4d9365403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a457a4beb4478b8a34e3633f8d2238" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 251.781 ms
time per row: 0.125891 ms
I20240502 21:25:48.464507 17715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:25:48.469893 17041 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:48.487432 17448 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:48.512789 17580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:48.533957 17312 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:25:48.567582 17175 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:48.815192 17041 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:48.818826 17715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:25:48.820636 17580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:48.824931 17448 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:48.848181 17312 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:25:48.850102 17175 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.157626 17041 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:49.175535 17715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:25:49.185321 17580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.189682 17175 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.200106 17448 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.202229 17312 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:25:49.454035 17041 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:49.458498 17715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:25:49.459738 17580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.476076 17448 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.476639 17312 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:25:49.493415 17175 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.806735 17041 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:49.808817 17448 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.811424 17715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:25:49.817063 17580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.822926 17175 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:49.826617 17312 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:25:50.301878 17041 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:50.304987 17580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:50.327925 17448 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:50.329102 17715 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:25:50.343979 17312 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:25:50.378715 17175 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
072464139bd54d21bce68f004b7f598d | 127.16.122.62:41017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.122.20:41201 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5a2cdeb7351f4fffb44146f4d9365403 | 127.16.122.1:40947 | HEALTHY | /A | 6 | 0
7181173c030d4a6db6e29c158fdb9a69 | 127.16.122.6:35119 | HEALTHY | /C | 4 | 0
86a457a4beb4478b8a34e3633f8d2238 | 127.16.122.5:36211 | HEALTHY | /C | 7 | 0
8c66e3c3cad649daa465f3ae946f95f7 | 127.16.122.4:36081 | HEALTHY | /B | 7 | 0
967c7c03fe754179ab1544965af0cf8d | 127.16.122.3:39535 | HEALTHY | /B | 5 | 0
ae4ceec4b9204f9987cc97cee5d1d820 | 127.16.122.2:45287 | HEALTHY | /A | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/B | 2
/A | 2
/C | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.16.122.1 | experimental | 127.16.122.1:40947
local_ip_for_outbound_sockets | 127.16.122.2 | experimental | 127.16.122.2:45287
local_ip_for_outbound_sockets | 127.16.122.3 | experimental | 127.16.122.3:39535
local_ip_for_outbound_sockets | 127.16.122.4 | experimental | 127.16.122.4:36081
local_ip_for_outbound_sockets | 127.16.122.5 | experimental | 127.16.122.5:36211
local_ip_for_outbound_sockets | 127.16.122.6 | experimental | 127.16.122.6:35119
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.122.1:40947
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.122.2:45287
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.122.3:39535
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.122.4:36081
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.122.5:36211
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.122.6:35119
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.122.20:41201 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:25:50.445469 16872 log_verifier.cc:126] Checking tablet 0070b68a1e5242e5a71ac1c7433c0425
I20240502 21:25:50.486582 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0070b68a1e5242e5a71ac1c7433c0425
I20240502 21:25:50.486789 16872 log_verifier.cc:126] Checking tablet 125cd0bebf43445f8f2d1d7a54f7c036
I20240502 21:25:50.524487 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 125cd0bebf43445f8f2d1d7a54f7c036
I20240502 21:25:50.524631 16872 log_verifier.cc:126] Checking tablet 1c5df765230c4f45bbd53d2e1e603aab
I20240502 21:25:50.567843 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1c5df765230c4f45bbd53d2e1e603aab
I20240502 21:25:50.568169 16872 log_verifier.cc:126] Checking tablet 1cbccee0bdc9476c99e9926db224c5b9
I20240502 21:25:50.609689 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1cbccee0bdc9476c99e9926db224c5b9
I20240502 21:25:50.609814 16872 log_verifier.cc:126] Checking tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f
I20240502 21:25:50.651711 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 22fb4c9d1de94ec6b8e1ab4eec11af9f
I20240502 21:25:50.651825 16872 log_verifier.cc:126] Checking tablet 2695b374d7fe417aa0a41ce5a06afa5d
I20240502 21:25:50.693305 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2695b374d7fe417aa0a41ce5a06afa5d
I20240502 21:25:50.693439 16872 log_verifier.cc:126] Checking tablet 27ff128fa4a245b59130a1c8150ed1c0
I20240502 21:25:50.732187 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 27ff128fa4a245b59130a1c8150ed1c0
I20240502 21:25:50.732303 16872 log_verifier.cc:126] Checking tablet 34b0d71e9d72459aaa0917655434cf47
I20240502 21:25:50.771009 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 34b0d71e9d72459aaa0917655434cf47
I20240502 21:25:50.771142 16872 log_verifier.cc:126] Checking tablet 36b6b065fc0c49cea3fac5df0fbdf096
I20240502 21:25:50.811208 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 36b6b065fc0c49cea3fac5df0fbdf096
I20240502 21:25:50.811323 16872 log_verifier.cc:126] Checking tablet 45d1c89db943440486f4f1efa57fcceb
I20240502 21:25:50.850958 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 45d1c89db943440486f4f1efa57fcceb
I20240502 21:25:50.851217 16872 log_verifier.cc:126] Checking tablet 4a18b7abc37b41bbb841dc9af174e035
I20240502 21:25:50.896869 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4a18b7abc37b41bbb841dc9af174e035
I20240502 21:25:50.897078 16872 log_verifier.cc:126] Checking tablet 52874dc477c0433984c9c59c583b8b20
I20240502 21:25:50.938059 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 52874dc477c0433984c9c59c583b8b20
I20240502 21:25:50.938220 16872 log_verifier.cc:126] Checking tablet 5d668e9869144eb88bd21b74eef3a6a2
I20240502 21:25:50.978320 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5d668e9869144eb88bd21b74eef3a6a2
I20240502 21:25:50.978488 16872 log_verifier.cc:126] Checking tablet 66416d27cc074e2a899d80a6bb444f87
I20240502 21:25:51.018075 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 66416d27cc074e2a899d80a6bb444f87
I20240502 21:25:51.018199 16872 log_verifier.cc:126] Checking tablet 674644cf151b4f0f89f7b037263cc9a8
I20240502 21:25:51.063926 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 674644cf151b4f0f89f7b037263cc9a8
I20240502 21:25:51.064069 16872 log_verifier.cc:126] Checking tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b
I20240502 21:25:51.115746 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6b3e5b8ed4fa4dbbb9d28453f7b55b7b
I20240502 21:25:51.115864 16872 log_verifier.cc:126] Checking tablet 6e770c72315d452292acca3919362b00
I20240502 21:25:51.164273 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6e770c72315d452292acca3919362b00
I20240502 21:25:51.164400 16872 log_verifier.cc:126] Checking tablet 7553a4d675be4edb9c42f655b8e439da
I20240502 21:25:51.208786 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7553a4d675be4edb9c42f655b8e439da
I20240502 21:25:51.208907 16872 log_verifier.cc:126] Checking tablet 7886bd3d3e2048c8900a130f8f516082
I20240502 21:25:51.251842 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7886bd3d3e2048c8900a130f8f516082
I20240502 21:25:51.252002 16872 log_verifier.cc:126] Checking tablet 8637ea61f53c47298f98d7e2b3f7d34f
I20240502 21:25:51.298159 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8637ea61f53c47298f98d7e2b3f7d34f
I20240502 21:25:51.298298 16872 log_verifier.cc:126] Checking tablet 8c15b912488947c0a3f29cc19e5fce61
I20240502 21:25:51.344239 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8c15b912488947c0a3f29cc19e5fce61
I20240502 21:25:51.344374 16872 log_verifier.cc:126] Checking tablet 8d59bf301e264f25b94f4bb93d4bfd82
I20240502 21:25:51.391120 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8d59bf301e264f25b94f4bb93d4bfd82
I20240502 21:25:51.391255 16872 log_verifier.cc:126] Checking tablet 8faf03c730e341b8bc3edb9a72d03d39
I20240502 21:25:51.437127 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8faf03c730e341b8bc3edb9a72d03d39
I20240502 21:25:51.437280 16872 log_verifier.cc:126] Checking tablet 9edf4032df9c47caa29d56ef153d85cc
I20240502 21:25:51.483739 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9edf4032df9c47caa29d56ef153d85cc
I20240502 21:25:51.483896 16872 log_verifier.cc:126] Checking tablet a27298e93e18457ca4cc29dc87b764c9
I20240502 21:25:51.528913 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a27298e93e18457ca4cc29dc87b764c9
I20240502 21:25:51.529055 16872 log_verifier.cc:126] Checking tablet bb83ef0f3aa84014afb1eb0e664e8ef3
I20240502 21:25:51.572203 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bb83ef0f3aa84014afb1eb0e664e8ef3
I20240502 21:25:51.572347 16872 log_verifier.cc:126] Checking tablet c7115fd506c540eaa3ad9c3904c410e8
I20240502 21:25:51.621311 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c7115fd506c540eaa3ad9c3904c410e8
I20240502 21:25:51.621449 16872 log_verifier.cc:126] Checking tablet c839ab1b86104fe6a1b6fbf7ddff1d19
I20240502 21:25:51.665669 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c839ab1b86104fe6a1b6fbf7ddff1d19
I20240502 21:25:51.665831 16872 log_verifier.cc:126] Checking tablet c8924e38adc84a4fb43d7c1daf41dce3
I20240502 21:25:51.709998 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c8924e38adc84a4fb43d7c1daf41dce3
I20240502 21:25:51.710134 16872 log_verifier.cc:126] Checking tablet cbfe08cd1c944e3f9a5438d204dfd196
I20240502 21:25:51.752305 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cbfe08cd1c944e3f9a5438d204dfd196
I20240502 21:25:51.752439 16872 log_verifier.cc:126] Checking tablet cd187707226242289cf3c2be45ccb955
I20240502 21:25:51.795428 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cd187707226242289cf3c2be45ccb955
I20240502 21:25:51.795573 16872 log_verifier.cc:126] Checking tablet ce5736b1f933416fb88223e3557cf57e
I20240502 21:25:51.836609 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ce5736b1f933416fb88223e3557cf57e
I20240502 21:25:51.836763 16872 log_verifier.cc:126] Checking tablet d684fc5640034615930845d0a908eb10
I20240502 21:25:51.879565 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d684fc5640034615930845d0a908eb10
I20240502 21:25:51.879693 16872 log_verifier.cc:126] Checking tablet e2d1434fc07c45f0be0a2015f8e31cae
I20240502 21:25:51.927592 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e2d1434fc07c45f0be0a2015f8e31cae
I20240502 21:25:51.927726 16872 log_verifier.cc:126] Checking tablet f0993ddf93d545cf8de9d26bd2085556
I20240502 21:25:51.973839 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f0993ddf93d545cf8de9d26bd2085556
I20240502 21:25:51.973996 16872 log_verifier.cc:126] Checking tablet ff283b6ba42f4d75b0092ddfc23af7a2
I20240502 21:25:52.025184 16872 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ff283b6ba42f4d75b0092ddfc23af7a2
I20240502 21:25:52.049422 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 16979
I20240502 21:25:52.067003 17103 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:52.215986 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 16979
W20240502 21:25:52.242880 17265 connection.cc:535] client connection to 127.16.122.1:40947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:52.243213 17265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:52.243536 17534 connection.cc:535] server connection from 127.16.122.1:38807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:52.243688 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 17112
W20240502 21:25:52.245319 17265 consensus_peers.cc:485] T 674644cf151b4f0f89f7b037263cc9a8 P 967c7c03fe754179ab1544965af0cf8d -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.245527 17265 consensus_peers.cc:485] T 27ff128fa4a245b59130a1c8150ed1c0 P 967c7c03fe754179ab1544965af0cf8d -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.245636 17265 consensus_peers.cc:485] T 66416d27cc074e2a899d80a6bb444f87 P 967c7c03fe754179ab1544965af0cf8d -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.250842 17238 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:25:52.290767 17534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111)
W20240502 21:25:52.292443 17534 consensus_peers.cc:485] T 6e770c72315d452292acca3919362b00 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.380887 17534 consensus_peers.cc:485] T 6b3e5b8ed4fa4dbbb9d28453f7b55b7b P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.392426 17534 consensus_peers.cc:485] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.395056 17265 consensus_peers.cc:485] T cbfe08cd1c944e3f9a5438d204dfd196 P 967c7c03fe754179ab1544965af0cf8d -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.398969 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 17112
W20240502 21:25:52.427716 17669 connection.cc:535] server connection from 127.16.122.2:43971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:52.428619 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 17248
W20240502 21:25:52.429071 17534 consensus_peers.cc:485] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.435649 17374 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:25:52.515625 17401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111)
W20240502 21:25:52.517998 17401 consensus_peers.cc:485] T 5d668e9869144eb88bd21b74eef3a6a2 P 8c66e3c3cad649daa465f3ae946f95f7 -> Peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Couldn't send request to peer ae4ceec4b9204f9987cc97cee5d1d820. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.518242 17668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111)
W20240502 21:25:52.519893 17668 consensus_peers.cc:485] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 -> Peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Couldn't send request to peer ae4ceec4b9204f9987cc97cee5d1d820. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.525672 17534 consensus_peers.cc:485] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.531281 17401 consensus_peers.cc:485] T ce5736b1f933416fb88223e3557cf57e P 8c66e3c3cad649daa465f3ae946f95f7 -> Peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Couldn't send request to peer ae4ceec4b9204f9987cc97cee5d1d820. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.546603 17534 consensus_peers.cc:485] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.565876 17668 consensus_peers.cc:485] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 -> Peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Couldn't send request to peer ae4ceec4b9204f9987cc97cee5d1d820. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.572196 17668 consensus_peers.cc:485] T 8637ea61f53c47298f98d7e2b3f7d34f P 7181173c030d4a6db6e29c158fdb9a69 -> Peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Couldn't send request to peer ae4ceec4b9204f9987cc97cee5d1d820. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.587031 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 17248
W20240502 21:25:52.593225 17668 consensus_peers.cc:485] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 -> Peer ae4ceec4b9204f9987cc97cee5d1d820 (127.16.122.2:45287): Couldn't send request to peer ae4ceec4b9204f9987cc97cee5d1d820. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.2:45287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.613015 17533 connection.cc:535] server connection from 127.16.122.3:44365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:52.613549 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 17383
W20240502 21:25:52.615635 17534 consensus_peers.cc:485] T 8c15b912488947c0a3f29cc19e5fce61 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Couldn't send request to peer 967c7c03fe754179ab1544965af0cf8d. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.3:39535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.617003 17534 consensus_peers.cc:485] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Couldn't send request to peer 967c7c03fe754179ab1544965af0cf8d. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.3:39535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.625702 17509 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:25:52.662539 17534 consensus_peers.cc:485] T 8d59bf301e264f25b94f4bb93d4bfd82 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Couldn't send request to peer 967c7c03fe754179ab1544965af0cf8d. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.3:39535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.665429 17534 consensus_peers.cc:485] T 22fb4c9d1de94ec6b8e1ab4eec11af9f P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Couldn't send request to peer 967c7c03fe754179ab1544965af0cf8d. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.3:39535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.672860 17534 consensus_peers.cc:485] T f0993ddf93d545cf8de9d26bd2085556 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 5a2cdeb7351f4fffb44146f4d9365403 (127.16.122.1:40947): Couldn't send request to peer 5a2cdeb7351f4fffb44146f4d9365403. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:40947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.766294 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 17383
W20240502 21:25:52.770218 17534 consensus_peers.cc:485] T 36b6b065fc0c49cea3fac5df0fbdf096 P 86a457a4beb4478b8a34e3633f8d2238 -> Peer 967c7c03fe754179ab1544965af0cf8d (127.16.122.3:39535): Couldn't send request to peer 967c7c03fe754179ab1544965af0cf8d. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.3:39535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.792191 17667 connection.cc:535] server connection from 127.16.122.4:40177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:52.792676 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 17517
W20240502 21:25:52.793474 17670 consensus_peers.cc:485] T 2695b374d7fe417aa0a41ce5a06afa5d P 7181173c030d4a6db6e29c158fdb9a69 -> Peer 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081): Couldn't send request to peer 8c66e3c3cad649daa465f3ae946f95f7. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.4:36081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:25:52.793682 17670 consensus_peers.cc:485] T c839ab1b86104fe6a1b6fbf7ddff1d19 P 7181173c030d4a6db6e29c158fdb9a69 -> Peer 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081): Couldn't send request to peer 8c66e3c3cad649daa465f3ae946f95f7. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.4:36081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.799433 17643 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:52.942559 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 17517
W20240502 21:25:52.966423 17670 consensus_peers.cc:485] T 7886bd3d3e2048c8900a130f8f516082 P 7181173c030d4a6db6e29c158fdb9a69 -> Peer 8c66e3c3cad649daa465f3ae946f95f7 (127.16.122.4:36081): Couldn't send request to peer 8c66e3c3cad649daa465f3ae946f95f7. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.4:36081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:52.969301 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 17651
I20240502 21:25:52.976248 17775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:53.118881 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 17651
I20240502 21:25:53.145356 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 16886
I20240502 21:25:53.146577 16949 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:53.310963 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 16886
2024-05-02T21:25:53Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (31081 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (31081 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 21:25:53.479199 16872 test_util.cc:268] Using random seed: 1064929773
I20240502 21:25:53.480994 16872 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:53.481081 16872 ts_itest-base.cc:113] --------------
I20240502 21:25:53.481129 16872 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:25:53.481202 16872 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:53.481258 16872 ts_itest-base.cc:116] --------------
2024-05-02T21:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:53Z Disabled control of system clock
I20240502 21:25:53.521735 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:45587
--webserver_interface=127.16.122.62
--webserver_port=0
--builtin_ntp_servers=127.16.122.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.62:45587
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:25:53.859510 19275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:53.859831 19275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:53.859927 19275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:53.867565 19275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:53.867698 19275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:53.867743 19275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:53.867803 19275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:53.867851 19275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:53.876986 19275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.122.62:45587
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.122.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:53.878381 19275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:53.880133 19275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:53.892411 19280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:53.893599 19281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:53.894588 19283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:53.895762 19275 server_base.cc:1034] running on GCE node
I20240502 21:25:53.896610 19275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:53.898198 19275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:53.899416 19275 hybrid_clock.cc:648] HybridClock initialized: now 1714685153899366 us; error 64 us; skew 500 ppm
I20240502 21:25:53.899788 19275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:53.902181 19275 webserver.cc:460] Webserver started at http://127.16.122.62:46755/ using document root <none> and password file <none>
I20240502 21:25:53.902966 19275 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:53.903086 19275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:53.903429 19275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:53.906137 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "45419e269ebe4c239612d6dc001172e8"
format_stamp: "Formatted at 2024-05-02 21:25:53 on dist-test-slave-p9xn"
I20240502 21:25:53.906996 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "45419e269ebe4c239612d6dc001172e8"
format_stamp: "Formatted at 2024-05-02 21:25:53 on dist-test-slave-p9xn"
I20240502 21:25:53.912309 19275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:53.915824 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:53.917210 19275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:53.917358 19275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
uuid: "45419e269ebe4c239612d6dc001172e8"
format_stamp: "Formatted at 2024-05-02 21:25:53 on dist-test-slave-p9xn"
I20240502 21:25:53.917551 19275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:53.931931 19275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:53.932715 19275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:53.932955 19275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:53.951309 19275 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.62:45587
I20240502 21:25:53.951344 19341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.62:45587 every 8 connection(s)
I20240502 21:25:53.952701 19275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:53.956418 19342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:53.957008 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 19275
I20240502 21:25:53.957208 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:53.969931 19342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8: Bootstrap starting.
I20240502 21:25:53.973251 19342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:53.974263 19342 log.cc:824] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:53.976826 19342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8: No bootstrap required, opened a new log
I20240502 21:25:53.984081 19342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45419e269ebe4c239612d6dc001172e8" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45587 } }
I20240502 21:25:53.984411 19342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:53.984495 19342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45419e269ebe4c239612d6dc001172e8, State: Initialized, Role: FOLLOWER
I20240502 21:25:53.985082 19342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45419e269ebe4c239612d6dc001172e8" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45587 } }
I20240502 21:25:53.985294 19342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:53.985432 19342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:53.985630 19342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:53.987769 19342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45419e269ebe4c239612d6dc001172e8" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45587 } }
I20240502 21:25:53.988250 19342 leader_election.cc:302] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45419e269ebe4c239612d6dc001172e8; no voters:
I20240502 21:25:53.988833 19342 leader_election.cc:288] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:53.988972 19348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:53.989667 19348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [term 1 LEADER]: Becoming Leader. State: Replica: 45419e269ebe4c239612d6dc001172e8, State: Running, Role: LEADER
I20240502 21:25:53.990165 19348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45419e269ebe4c239612d6dc001172e8" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45587 } }
I20240502 21:25:53.991122 19342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:53.993224 19350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45419e269ebe4c239612d6dc001172e8. Latest consensus state: current_term: 1 leader_uuid: "45419e269ebe4c239612d6dc001172e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45419e269ebe4c239612d6dc001172e8" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45587 } } }
I20240502 21:25:53.993294 19349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45419e269ebe4c239612d6dc001172e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45419e269ebe4c239612d6dc001172e8" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45587 } } }
I20240502 21:25:53.993597 19350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:53.993644 19349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:53.994860 19353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:54.000595 19353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:54.019492 19353 catalog_manager.cc:1347] Generated new cluster ID: 0a142148787a423b8702c50b82951047
I20240502 21:25:54.019627 19353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:54.029731 19353 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:54.031628 19353 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:54.049096 19353 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 45419e269ebe4c239612d6dc001172e8: Generated new TSK 0
I20240502 21:25:54.050271 19353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:54.058701 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.1:0
--local_ip_for_outbound_sockets=127.16.122.1
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--builtin_ntp_servers=127.16.122.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:54.407085 19367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:54.407400 19367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:54.407505 19367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:54.415115 19367 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:54.415252 19367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:54.415374 19367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.1
I20240502 21:25:54.425473 19367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.1
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:54.426858 19367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:54.428588 19367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:54.441439 19372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:54.442625 19373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:54.444099 19375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:54.445192 19367 server_base.cc:1034] running on GCE node
I20240502 21:25:54.445866 19367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:54.446910 19367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:54.448143 19367 hybrid_clock.cc:648] HybridClock initialized: now 1714685154448087 us; error 43 us; skew 500 ppm
I20240502 21:25:54.448490 19367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:54.450888 19367 webserver.cc:460] Webserver started at http://127.16.122.1:34285/ using document root <none> and password file <none>
I20240502 21:25:54.451660 19367 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:54.451786 19367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:54.452116 19367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:54.454973 19367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "06f47dd8dead4768880f0d42e6fc7ac9"
format_stamp: "Formatted at 2024-05-02 21:25:54 on dist-test-slave-p9xn"
I20240502 21:25:54.455821 19367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "06f47dd8dead4768880f0d42e6fc7ac9"
format_stamp: "Formatted at 2024-05-02 21:25:54 on dist-test-slave-p9xn"
I20240502 21:25:54.461539 19367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:54.465269 19381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:54.466809 19367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 21:25:54.466987 19367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "06f47dd8dead4768880f0d42e6fc7ac9"
format_stamp: "Formatted at 2024-05-02 21:25:54 on dist-test-slave-p9xn"
I20240502 21:25:54.467231 19367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:54.489845 19367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:54.490913 19367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:54.491219 19367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:54.492373 19367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:54.494114 19367 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:54.494228 19367 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:54.494324 19367 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:54.494421 19367 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 21:25:54.527981 19367 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.1:43835
I20240502 21:25:54.528033 19495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.1:43835 every 8 connection(s)
I20240502 21:25:54.529403 19367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:54.536010 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 19367
I20240502 21:25:54.536244 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:54.540117 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.2:0
--local_ip_for_outbound_sockets=127.16.122.2
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--builtin_ntp_servers=127.16.122.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:54.544824 19496 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45587
I20240502 21:25:54.545168 19496 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:54.546084 19496 heartbeater.cc:511] Master 127.16.122.62:45587 requested a full tablet report, sending...
I20240502 21:25:54.548717 19306 ts_manager.cc:194] Registered new tserver with Master: 06f47dd8dead4768880f0d42e6fc7ac9 (127.16.122.1:43835)
I20240502 21:25:54.550374 19306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.1:33523
W20240502 21:25:54.887095 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:54.887400 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:54.887531 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:54.895160 19500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:54.895298 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:54.895462 19500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.2
I20240502 21:25:54.904527 19500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.2
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:54.906071 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:54.907861 19500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:54.921135 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:54.922251 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:54.923290 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:54.924443 19500 server_base.cc:1034] running on GCE node
I20240502 21:25:54.925155 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:54.926211 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:54.927469 19500 hybrid_clock.cc:648] HybridClock initialized: now 1714685154927404 us; error 37 us; skew 500 ppm
I20240502 21:25:54.927852 19500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:54.930439 19500 webserver.cc:460] Webserver started at http://127.16.122.2:41421/ using document root <none> and password file <none>
I20240502 21:25:54.931231 19500 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:54.931345 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:54.931681 19500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:54.934341 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dd8480b1586e45558201e18ca238cedc"
format_stamp: "Formatted at 2024-05-02 21:25:54 on dist-test-slave-p9xn"
I20240502 21:25:54.935323 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dd8480b1586e45558201e18ca238cedc"
format_stamp: "Formatted at 2024-05-02 21:25:54 on dist-test-slave-p9xn"
I20240502 21:25:54.941002 19500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 21:25:54.944676 19514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:54.946200 19500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:54.946429 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dd8480b1586e45558201e18ca238cedc"
format_stamp: "Formatted at 2024-05-02 21:25:54 on dist-test-slave-p9xn"
I20240502 21:25:54.946647 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:54.972131 19500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:54.973114 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:54.973415 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:54.974498 19500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:54.976078 19500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:54.976188 19500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:54.976303 19500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:54.976375 19500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:55.010341 19500 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.2:43911
I20240502 21:25:55.010423 19628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.2:43911 every 8 connection(s)
I20240502 21:25:55.011713 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:55.019486 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 19500
I20240502 21:25:55.019673 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:55.023152 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.3:0
--local_ip_for_outbound_sockets=127.16.122.3
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--builtin_ntp_servers=127.16.122.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:55.025030 19629 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45587
I20240502 21:25:55.025453 19629 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:55.026645 19629 heartbeater.cc:511] Master 127.16.122.62:45587 requested a full tablet report, sending...
I20240502 21:25:55.028370 19306 ts_manager.cc:194] Registered new tserver with Master: dd8480b1586e45558201e18ca238cedc (127.16.122.2:43911)
I20240502 21:25:55.029150 19306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.2:59093
W20240502 21:25:55.364111 19632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:55.364454 19632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:55.364578 19632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:55.372278 19632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:55.372432 19632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:55.372601 19632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.3
I20240502 21:25:55.381927 19632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.3
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:55.383350 19632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:55.385080 19632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:55.398032 19637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:55.399080 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:55.399987 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:55.401031 19632 server_base.cc:1034] running on GCE node
I20240502 21:25:55.401625 19632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:55.402631 19632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:55.403836 19632 hybrid_clock.cc:648] HybridClock initialized: now 1714685155403788 us; error 45 us; skew 500 ppm
I20240502 21:25:55.404135 19632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:55.406296 19632 webserver.cc:460] Webserver started at http://127.16.122.3:43733/ using document root <none> and password file <none>
I20240502 21:25:55.407047 19632 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:55.407161 19632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:55.407496 19632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:55.410104 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "346f8b24f65d431cba0a8b0e3c4280b0"
format_stamp: "Formatted at 2024-05-02 21:25:55 on dist-test-slave-p9xn"
I20240502 21:25:55.410897 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "346f8b24f65d431cba0a8b0e3c4280b0"
format_stamp: "Formatted at 2024-05-02 21:25:55 on dist-test-slave-p9xn"
I20240502 21:25:55.416594 19632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240502 21:25:55.420441 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:55.421917 19632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:25:55.422082 19632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "346f8b24f65d431cba0a8b0e3c4280b0"
format_stamp: "Formatted at 2024-05-02 21:25:55 on dist-test-slave-p9xn"
I20240502 21:25:55.422277 19632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:55.439249 19632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:55.440109 19632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:55.440400 19632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:55.441318 19632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:55.442822 19632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:55.442922 19632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:55.443039 19632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:55.443116 19632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:55.475847 19632 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.3:37787
I20240502 21:25:55.475903 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.3:37787 every 8 connection(s)
I20240502 21:25:55.477234 19632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:55.481102 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 19632
I20240502 21:25:55.481283 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:55.484477 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.4:0
--local_ip_for_outbound_sockets=127.16.122.4
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--builtin_ntp_servers=127.16.122.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:55.491012 19759 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45587
I20240502 21:25:55.491397 19759 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:55.492467 19759 heartbeater.cc:511] Master 127.16.122.62:45587 requested a full tablet report, sending...
I20240502 21:25:55.494244 19306 ts_manager.cc:194] Registered new tserver with Master: 346f8b24f65d431cba0a8b0e3c4280b0 (127.16.122.3:37787)
I20240502 21:25:55.495492 19306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.3:39693
I20240502 21:25:55.553611 19496 heartbeater.cc:503] Master 127.16.122.62:45587 was elected leader, sending a full tablet report...
W20240502 21:25:55.838793 19762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:55.839138 19762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:55.839291 19762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:55.847206 19762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:55.847342 19762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:55.847520 19762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.4
I20240502 21:25:55.857059 19762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.4
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:55.858593 19762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:55.860484 19762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:55.874328 19767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:55.875449 19768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:55.876438 19770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:55.877521 19762 server_base.cc:1034] running on GCE node
I20240502 21:25:55.878219 19762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:55.879340 19762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:55.880568 19762 hybrid_clock.cc:648] HybridClock initialized: now 1714685155880526 us; error 33 us; skew 500 ppm
I20240502 21:25:55.880934 19762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:55.883481 19762 webserver.cc:460] Webserver started at http://127.16.122.4:43497/ using document root <none> and password file <none>
I20240502 21:25:55.884263 19762 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:55.884375 19762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:55.884711 19762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:55.887456 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5c45687f16d74e8e9cad381008c8c5e4"
format_stamp: "Formatted at 2024-05-02 21:25:55 on dist-test-slave-p9xn"
I20240502 21:25:55.888237 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5c45687f16d74e8e9cad381008c8c5e4"
format_stamp: "Formatted at 2024-05-02 21:25:55 on dist-test-slave-p9xn"
I20240502 21:25:55.893577 19762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:25:55.897152 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:55.898578 19762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:55.898742 19762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5c45687f16d74e8e9cad381008c8c5e4"
format_stamp: "Formatted at 2024-05-02 21:25:55 on dist-test-slave-p9xn"
I20240502 21:25:55.898945 19762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:55.963843 19762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:55.964821 19762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:55.965122 19762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:55.966168 19762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:55.967830 19762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:55.967937 19762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:55.968065 19762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:55.968139 19762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.006546 19762 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.4:36167
I20240502 21:25:56.006596 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.4:36167 every 8 connection(s)
I20240502 21:25:56.007902 19762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:56.014499 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 19762
I20240502 21:25:56.014703 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:56.018209 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.5:0
--local_ip_for_outbound_sockets=127.16.122.5
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--builtin_ntp_servers=127.16.122.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:56.024567 19889 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45587
I20240502 21:25:56.024868 19889 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:56.025667 19889 heartbeater.cc:511] Master 127.16.122.62:45587 requested a full tablet report, sending...
I20240502 21:25:56.027124 19306 ts_manager.cc:194] Registered new tserver with Master: 5c45687f16d74e8e9cad381008c8c5e4 (127.16.122.4:36167)
I20240502 21:25:56.027978 19306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.4:46331
I20240502 21:25:56.032354 19629 heartbeater.cc:503] Master 127.16.122.62:45587 was elected leader, sending a full tablet report...
I20240502 21:25:56.498458 19759 heartbeater.cc:503] Master 127.16.122.62:45587 was elected leader, sending a full tablet report...
W20240502 21:25:56.669345 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:56.669952 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:56.670286 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:56.687750 19893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:56.688050 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:56.688266 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.5
I20240502 21:25:56.699064 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45587
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.5
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:56.700546 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:56.702582 19893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:56.716430 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:56.717175 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:56.718410 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:56.719502 19893 server_base.cc:1034] running on GCE node
I20240502 21:25:56.720153 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:56.721241 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:56.722445 19893 hybrid_clock.cc:648] HybridClock initialized: now 1714685156722376 us; error 63 us; skew 500 ppm
I20240502 21:25:56.722828 19893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:56.725317 19893 webserver.cc:460] Webserver started at http://127.16.122.5:44891/ using document root <none> and password file <none>
I20240502 21:25:56.726101 19893 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:56.726233 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:56.726636 19893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:56.729393 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d1e8892953c14b368f3bee54a90905da"
format_stamp: "Formatted at 2024-05-02 21:25:56 on dist-test-slave-p9xn"
I20240502 21:25:56.730249 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d1e8892953c14b368f3bee54a90905da"
format_stamp: "Formatted at 2024-05-02 21:25:56 on dist-test-slave-p9xn"
I20240502 21:25:56.735951 19893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 21:25:56.739627 19907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.741200 19893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:25:56.741369 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1e8892953c14b368f3bee54a90905da"
format_stamp: "Formatted at 2024-05-02 21:25:56 on dist-test-slave-p9xn"
I20240502 21:25:56.741614 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:56.761642 19893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:56.762699 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:56.762995 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:56.764110 19893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:56.765782 19893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:56.765870 19893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.765980 19893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:56.766047 19893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.799779 19893 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.5:44969
I20240502 21:25:56.799841 20020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.5:44969 every 8 connection(s)
I20240502 21:25:56.801192 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:56.810101 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 19893
I20240502 21:25:56.810303 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:56.815029 20021 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45587
I20240502 21:25:56.815378 20021 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:56.816233 20021 heartbeater.cc:511] Master 127.16.122.62:45587 requested a full tablet report, sending...
I20240502 21:25:56.817616 19306 ts_manager.cc:194] Registered new tserver with Master: d1e8892953c14b368f3bee54a90905da (127.16.122.5:44969)
I20240502 21:25:56.818432 19306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.5:49957
I20240502 21:25:56.826848 16872 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:56.844014 19306 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50344:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:56.892413 19429 tablet_service.cc:1467] Processing CreateTablet for tablet 35caa075a1c34156b4f85d161f9a934d (DEFAULT_TABLE table=TestTable [id=c6773124007b48549afbf441d916b4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:56.892439 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 35caa075a1c34156b4f85d161f9a934d (DEFAULT_TABLE table=TestTable [id=c6773124007b48549afbf441d916b4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:56.892441 19563 tablet_service.cc:1467] Processing CreateTablet for tablet 35caa075a1c34156b4f85d161f9a934d (DEFAULT_TABLE table=TestTable [id=c6773124007b48549afbf441d916b4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:56.893756 19429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35caa075a1c34156b4f85d161f9a934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:56.893843 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35caa075a1c34156b4f85d161f9a934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:56.894083 19563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35caa075a1c34156b4f85d161f9a934d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:56.904448 20040 tablet_bootstrap.cc:492] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: Bootstrap starting.
I20240502 21:25:56.907410 20042 tablet_bootstrap.cc:492] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: Bootstrap starting.
I20240502 21:25:56.909236 20041 tablet_bootstrap.cc:492] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: Bootstrap starting.
I20240502 21:25:56.910187 20040 tablet_bootstrap.cc:654] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:56.911015 20042 tablet_bootstrap.cc:654] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:56.911623 20040 log.cc:824] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:56.912331 20042 log.cc:824] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:56.912817 20041 tablet_bootstrap.cc:654] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:56.913862 20041 log.cc:824] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:56.915093 20040 tablet_bootstrap.cc:492] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: No bootstrap required, opened a new log
I20240502 21:25:56.915391 20040 ts_tablet_manager.cc:1367] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 21:25:56.916834 20041 tablet_bootstrap.cc:492] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: No bootstrap required, opened a new log
I20240502 21:25:56.916982 20042 tablet_bootstrap.cc:492] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: No bootstrap required, opened a new log
I20240502 21:25:56.917104 20041 ts_tablet_manager.cc:1367] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 21:25:56.917279 20042 ts_tablet_manager.cc:1367] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20240502 21:25:56.924237 20042 raft_consensus.cc:348] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.924618 20042 raft_consensus.cc:374] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:56.924744 20042 raft_consensus.cc:729] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c45687f16d74e8e9cad381008c8c5e4, State: Initialized, Role: FOLLOWER
I20240502 21:25:56.925138 20040 raft_consensus.cc:348] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.925390 20042 consensus_queue.cc:260] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.925609 20040 raft_consensus.cc:374] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:56.925726 20040 raft_consensus.cc:729] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f47dd8dead4768880f0d42e6fc7ac9, State: Initialized, Role: FOLLOWER
I20240502 21:25:56.926110 20041 raft_consensus.cc:348] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.926584 20041 raft_consensus.cc:374] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:56.926541 20040 consensus_queue.cc:260] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.926687 20041 raft_consensus.cc:729] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8480b1586e45558201e18ca238cedc, State: Initialized, Role: FOLLOWER
I20240502 21:25:56.927559 20041 consensus_queue.cc:260] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.928500 20040 ts_tablet_manager.cc:1398] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20240502 21:25:56.929437 19889 heartbeater.cc:503] Master 127.16.122.62:45587 was elected leader, sending a full tablet report...
I20240502 21:25:56.930109 20042 ts_tablet_manager.cc:1398] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20240502 21:25:56.931327 20041 ts_tablet_manager.cc:1398] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20240502 21:25:56.996521 20048 raft_consensus.cc:482] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:56.996846 20048 raft_consensus.cc:504] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:56.999270 20048 leader_election.cc:288] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8480b1586e45558201e18ca238cedc (127.16.122.2:43911), 5c45687f16d74e8e9cad381008c8c5e4 (127.16.122.4:36167)
I20240502 21:25:57.007167 19584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35caa075a1c34156b4f85d161f9a934d" candidate_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8480b1586e45558201e18ca238cedc" is_pre_election: true
I20240502 21:25:57.007786 19584 raft_consensus.cc:2454] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f47dd8dead4768880f0d42e6fc7ac9 in term 0.
I20240502 21:25:57.008570 19385 leader_election.cc:302] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f47dd8dead4768880f0d42e6fc7ac9, dd8480b1586e45558201e18ca238cedc; no voters:
I20240502 21:25:57.009023 20048 raft_consensus.cc:2781] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:57.009202 20048 raft_consensus.cc:482] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:57.009333 20048 raft_consensus.cc:3037] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:57.009276 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35caa075a1c34156b4f85d161f9a934d" candidate_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c45687f16d74e8e9cad381008c8c5e4" is_pre_election: true
I20240502 21:25:57.009837 19844 raft_consensus.cc:2454] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f47dd8dead4768880f0d42e6fc7ac9 in term 0.
I20240502 21:25:57.011931 20048 raft_consensus.cc:504] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:57.012532 20048 leader_election.cc:288] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [CANDIDATE]: Term 1 election: Requested vote from peers dd8480b1586e45558201e18ca238cedc (127.16.122.2:43911), 5c45687f16d74e8e9cad381008c8c5e4 (127.16.122.4:36167)
W20240502 21:25:57.012974 19890 tablet.cc:2369] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:57.012979 19584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35caa075a1c34156b4f85d161f9a934d" candidate_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8480b1586e45558201e18ca238cedc"
I20240502 21:25:57.013198 19584 raft_consensus.cc:3037] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:25:57.015174 19630 tablet.cc:2369] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:57.016230 19584 raft_consensus.cc:2454] T 35caa075a1c34156b4f85d161f9a934d P dd8480b1586e45558201e18ca238cedc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f47dd8dead4768880f0d42e6fc7ac9 in term 1.
I20240502 21:25:57.016783 19385 leader_election.cc:302] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f47dd8dead4768880f0d42e6fc7ac9, dd8480b1586e45558201e18ca238cedc; no voters:
I20240502 21:25:57.016839 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35caa075a1c34156b4f85d161f9a934d" candidate_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c45687f16d74e8e9cad381008c8c5e4"
I20240502 21:25:57.017033 19844 raft_consensus.cc:3037] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:57.017107 20048 raft_consensus.cc:2781] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:57.019485 20048 raft_consensus.cc:686] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 06f47dd8dead4768880f0d42e6fc7ac9, State: Running, Role: LEADER
I20240502 21:25:57.020380 20048 consensus_queue.cc:237] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } }
I20240502 21:25:57.020771 19844 raft_consensus.cc:2454] T 35caa075a1c34156b4f85d161f9a934d P 5c45687f16d74e8e9cad381008c8c5e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f47dd8dead4768880f0d42e6fc7ac9 in term 1.
I20240502 21:25:57.026233 19306 catalog_manager.cc:5484] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06f47dd8dead4768880f0d42e6fc7ac9 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f47dd8dead4768880f0d42e6fc7ac9" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:57.033411 19497 tablet.cc:2369] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:57.071172 16872 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:57.073707 16872 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06f47dd8dead4768880f0d42e6fc7ac9 to finish bootstrapping
I20240502 21:25:57.082959 16872 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dd8480b1586e45558201e18ca238cedc to finish bootstrapping
I20240502 21:25:57.091010 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 346f8b24f65d431cba0a8b0e3c4280b0 to finish bootstrapping
I20240502 21:25:57.100189 16872 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5c45687f16d74e8e9cad381008c8c5e4 to finish bootstrapping
I20240502 21:25:57.108141 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d1e8892953c14b368f3bee54a90905da to finish bootstrapping
I20240502 21:25:57.564438 20079 consensus_queue.cc:1035] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c45687f16d74e8e9cad381008c8c5e4" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:57.581458 20079 consensus_queue.cc:1035] T 35caa075a1c34156b4f85d161f9a934d P 06f47dd8dead4768880f0d42e6fc7ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8480b1586e45558201e18ca238cedc" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:57.609999 19694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.623662 19824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.626799 19429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:57.652715 19955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.676226 19563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.809671 19429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:57.820845 20021 heartbeater.cc:503] Master 127.16.122.62:45587 was elected leader, sending a full tablet report...
I20240502 21:25:57.828022 19824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.828758 19955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.828821 19694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.850875 19563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.470151 19694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.471977 19824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.481616 19955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.491744 19429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:58.499600 19563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.660230 19955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.661305 19429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:58.661546 19694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.663700 19824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.680769 19563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.840227 19429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:58.850664 19824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.853404 19694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.855701 19955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:58.862259 19563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:59.029902 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 19367
I20240502 21:25:59.031764 19491 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:59.132337 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 19367
W20240502 21:25:59.152355 19293 connection.cc:535] server connection from 127.16.122.1:33523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:59.152500 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 19500
I20240502 21:25:59.153661 19624 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:59.251091 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 19500
I20240502 21:25:59.270677 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 19632
I20240502 21:25:59.271793 19753 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:59.367434 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 19632
I20240502 21:25:59.387244 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 19762
I20240502 21:25:59.388469 19883 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:59.487468 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 19762
I20240502 21:25:59.507190 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 19893
I20240502 21:25:59.508306 20015 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:59.605329 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 19893
I20240502 21:25:59.623597 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 19275
I20240502 21:25:59.624796 19337 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:59.745227 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 19275
2024-05-02T21:25:59Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (6313 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6313 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240502 21:25:59.792402 16872 test_util.cc:268] Using random seed: 1071242997
I20240502 21:25:59.794548 16872 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:59.794620 16872 ts_itest-base.cc:113] --------------
I20240502 21:25:59.794668 16872 ts_itest-base.cc:114] 7 tablet servers
I20240502 21:25:59.794726 16872 ts_itest-base.cc:115] 2 replicas per TS
I20240502 21:25:59.794793 16872 ts_itest-base.cc:116] --------------
2024-05-02T21:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:59Z Disabled control of system clock
I20240502 21:25:59.839995 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:40689
--webserver_interface=127.16.122.62
--webserver_port=0
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.62:40689
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 21:26:00.209345 20256 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:26:00.209710 20256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:00.209798 20256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:00.209875 20256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:00.217521 20256 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:00.217684 20256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:00.217756 20256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:00.217804 20256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:00.217865 20256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:00.227286 20256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.122.62:40689
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.122.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:00.228685 20256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:00.230494 20256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:00.241813 20261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:00.242753 20262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:00.243897 20264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:00.244957 20256 server_base.cc:1034] running on GCE node
I20240502 21:26:00.245806 20256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:00.247406 20256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:00.248648 20256 hybrid_clock.cc:648] HybridClock initialized: now 1714685160248579 us; error 43 us; skew 500 ppm
I20240502 21:26:00.248992 20256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:00.251408 20256 webserver.cc:460] Webserver started at http://127.16.122.62:36381/ using document root <none> and password file <none>
I20240502 21:26:00.252154 20256 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:00.252298 20256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:00.252614 20256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:00.255481 20256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ff826cbe705f48c5b303117140a0e6d2"
format_stamp: "Formatted at 2024-05-02 21:26:00 on dist-test-slave-p9xn"
I20240502 21:26:00.256243 20256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ff826cbe705f48c5b303117140a0e6d2"
format_stamp: "Formatted at 2024-05-02 21:26:00 on dist-test-slave-p9xn"
I20240502 21:26:00.261737 20256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 21:26:00.265312 20270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:00.266793 20256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:00.266973 20256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ff826cbe705f48c5b303117140a0e6d2"
format_stamp: "Formatted at 2024-05-02 21:26:00 on dist-test-slave-p9xn"
I20240502 21:26:00.267179 20256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:00.282943 20256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:00.283831 20256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:00.284122 20256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:00.303965 20256 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.62:40689
I20240502 21:26:00.304009 20322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.62:40689 every 8 connection(s)
I20240502 21:26:00.305476 20256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:26:00.307806 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 20256
I20240502 21:26:00.308040 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:26:00.310329 20323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:00.323357 20323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2: Bootstrap starting.
I20240502 21:26:00.326514 20323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:00.327554 20323 log.cc:824] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:00.330328 20323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2: No bootstrap required, opened a new log
I20240502 21:26:00.336480 20323 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff826cbe705f48c5b303117140a0e6d2" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 40689 } }
I20240502 21:26:00.336849 20323 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:00.336951 20323 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff826cbe705f48c5b303117140a0e6d2, State: Initialized, Role: FOLLOWER
I20240502 21:26:00.337599 20323 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff826cbe705f48c5b303117140a0e6d2" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 40689 } }
I20240502 21:26:00.337846 20323 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:00.337980 20323 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:00.338140 20323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:00.340492 20323 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff826cbe705f48c5b303117140a0e6d2" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 40689 } }
I20240502 21:26:00.341001 20323 leader_election.cc:302] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff826cbe705f48c5b303117140a0e6d2; no voters:
I20240502 21:26:00.341634 20323 leader_election.cc:288] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:00.341787 20328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:00.342459 20328 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [term 1 LEADER]: Becoming Leader. State: Replica: ff826cbe705f48c5b303117140a0e6d2, State: Running, Role: LEADER
I20240502 21:26:00.343065 20328 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff826cbe705f48c5b303117140a0e6d2" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 40689 } }
I20240502 21:26:00.344008 20323 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:00.345950 20330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff826cbe705f48c5b303117140a0e6d2. Latest consensus state: current_term: 1 leader_uuid: "ff826cbe705f48c5b303117140a0e6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff826cbe705f48c5b303117140a0e6d2" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 40689 } } }
I20240502 21:26:00.346268 20330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:00.347159 20329 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff826cbe705f48c5b303117140a0e6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff826cbe705f48c5b303117140a0e6d2" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 40689 } } }
I20240502 21:26:00.347628 20329 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2 [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:00.350960 20333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:00.356986 20333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:00.366852 20333 catalog_manager.cc:1347] Generated new cluster ID: dc6add13065447d5840939ae60df05ce
I20240502 21:26:00.366988 20333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:00.400429 20333 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:00.402283 20333 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:00.412823 20333 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ff826cbe705f48c5b303117140a0e6d2: Generated new TSK 0
I20240502 21:26:00.413789 20333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:00.420045 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.1:0
--local_ip_for_outbound_sockets=127.16.122.1
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:26:00.773608 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:00.773911 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:00.774008 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:00.781608 20347 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:00.781744 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:00.781862 20347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.1
I20240502 21:26:00.791767 20347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.1
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:00.793198 20347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:00.795115 20347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:00.808123 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:00.809363 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:00.810549 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:00.811723 20347 server_base.cc:1034] running on GCE node
I20240502 21:26:00.812479 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:00.813584 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:00.814818 20347 hybrid_clock.cc:648] HybridClock initialized: now 1714685160814773 us; error 42 us; skew 500 ppm
I20240502 21:26:00.815184 20347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:00.817685 20347 webserver.cc:460] Webserver started at http://127.16.122.1:36931/ using document root <none> and password file <none>
I20240502 21:26:00.818531 20347 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:00.818658 20347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:00.818993 20347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:00.821823 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "35812e2216cb4758aee1e94877a823b7"
format_stamp: "Formatted at 2024-05-02 21:26:00 on dist-test-slave-p9xn"
I20240502 21:26:00.822691 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "35812e2216cb4758aee1e94877a823b7"
format_stamp: "Formatted at 2024-05-02 21:26:00 on dist-test-slave-p9xn"
I20240502 21:26:00.828018 20347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:00.831729 20361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:00.833150 20347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:00.833317 20347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "35812e2216cb4758aee1e94877a823b7"
format_stamp: "Formatted at 2024-05-02 21:26:00 on dist-test-slave-p9xn"
I20240502 21:26:00.833508 20347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:00.852144 20347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:00.852984 20347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:00.853238 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:00.854148 20347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:00.856038 20347 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:00.856151 20347 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:00.856259 20347 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:00.856323 20347 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:00.890105 20347 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.1:39539
I20240502 21:26:00.890148 20473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.1:39539 every 8 connection(s)
I20240502 21:26:00.891742 20347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:00.901438 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 20347
I20240502 21:26:00.901672 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:26:00.905400 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.2:0
--local_ip_for_outbound_sockets=127.16.122.2
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:00.906250 20475 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:00.906586 20475 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:00.907486 20475 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:00.909945 20287 ts_manager.cc:194] Registered new tserver with Master: 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:00.911780 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.1:40847
W20240502 21:26:01.265704 20479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:01.266023 20479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:01.266139 20479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:01.273995 20479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:01.274125 20479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:01.274289 20479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.2
I20240502 21:26:01.283427 20479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.2
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:01.284821 20479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:01.286628 20479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:01.299826 20484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:01.301246 20485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:01.301728 20487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:01.303011 20479 server_base.cc:1034] running on GCE node
I20240502 21:26:01.303772 20479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:01.304953 20479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:01.306236 20479 hybrid_clock.cc:648] HybridClock initialized: now 1714685161306184 us; error 42 us; skew 500 ppm
I20240502 21:26:01.306655 20479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:01.309294 20479 webserver.cc:460] Webserver started at http://127.16.122.2:46155/ using document root <none> and password file <none>
I20240502 21:26:01.310138 20479 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:01.310259 20479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:01.310654 20479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:01.313405 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "654f8d3e16bb4a658607a5c26f261cbf"
format_stamp: "Formatted at 2024-05-02 21:26:01 on dist-test-slave-p9xn"
I20240502 21:26:01.314249 20479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "654f8d3e16bb4a658607a5c26f261cbf"
format_stamp: "Formatted at 2024-05-02 21:26:01 on dist-test-slave-p9xn"
I20240502 21:26:01.319896 20479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 21:26:01.323580 20493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:01.325012 20479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:01.325184 20479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "654f8d3e16bb4a658607a5c26f261cbf"
format_stamp: "Formatted at 2024-05-02 21:26:01 on dist-test-slave-p9xn"
I20240502 21:26:01.325395 20479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:01.344285 20479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:01.345178 20479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:01.345461 20479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:01.346479 20479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:01.347968 20479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:01.348057 20479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:01.348186 20479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:01.348256 20479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:01.381326 20479 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.2:40149
I20240502 21:26:01.381376 20611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.2:40149 every 8 connection(s)
I20240502 21:26:01.382769 20479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:26:01.383702 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 20479
I20240502 21:26:01.383886 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:26:01.387853 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.3:0
--local_ip_for_outbound_sockets=127.16.122.3
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:01.399859 20612 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:01.400188 20612 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:01.401043 20612 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:01.402534 20287 ts_manager.cc:194] Registered new tserver with Master: 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:01.403318 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.2:39831
W20240502 21:26:01.728987 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:01.729368 20616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:01.729517 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:01.736888 20616 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:01.737025 20616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:01.737219 20616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.3
I20240502 21:26:01.746428 20616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.3
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:01.747809 20616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:01.749694 20616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:01.762737 20621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:01.763736 20624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:01.764276 20622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:01.764878 20616 server_base.cc:1034] running on GCE node
I20240502 21:26:01.765529 20616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:01.766558 20616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:01.767757 20616 hybrid_clock.cc:648] HybridClock initialized: now 1714685161767708 us; error 40 us; skew 500 ppm
I20240502 21:26:01.768078 20616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:01.770514 20616 webserver.cc:460] Webserver started at http://127.16.122.3:35379/ using document root <none> and password file <none>
I20240502 21:26:01.771258 20616 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:01.771363 20616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:01.771698 20616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:01.774303 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0ac8c8b961dc4500bc8b9f1055298244"
format_stamp: "Formatted at 2024-05-02 21:26:01 on dist-test-slave-p9xn"
I20240502 21:26:01.775084 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0ac8c8b961dc4500bc8b9f1055298244"
format_stamp: "Formatted at 2024-05-02 21:26:01 on dist-test-slave-p9xn"
I20240502 21:26:01.780158 20616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 21:26:01.783610 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:01.784895 20616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:01.785049 20616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0ac8c8b961dc4500bc8b9f1055298244"
format_stamp: "Formatted at 2024-05-02 21:26:01 on dist-test-slave-p9xn"
I20240502 21:26:01.785264 20616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:01.805855 20616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:01.806844 20616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:01.807108 20616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:01.808077 20616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:01.809615 20616 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:01.809702 20616 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:01.809787 20616 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:01.809841 20616 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:01.842844 20616 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.3:34769
I20240502 21:26:01.842914 20743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.3:34769 every 8 connection(s)
I20240502 21:26:01.844197 20616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:26:01.847157 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 20616
I20240502 21:26:01.847373 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:26:01.850339 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.4:0
--local_ip_for_outbound_sockets=127.16.122.4
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:01.857488 20744 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:01.857823 20744 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:01.858738 20744 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:01.860219 20287 ts_manager.cc:194] Registered new tserver with Master: 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:01.861043 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.3:33855
I20240502 21:26:01.914886 20475 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
W20240502 21:26:02.200968 20747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:02.201328 20747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:02.201484 20747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:02.209111 20747 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:02.209273 20747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:02.209450 20747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.4
I20240502 21:26:02.218708 20747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.4
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:02.220129 20747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:02.221892 20747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:02.234673 20752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:02.235847 20747 server_base.cc:1034] running on GCE node
W20240502 21:26:02.236891 20753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:02.237824 20755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:02.238664 20747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:02.239698 20747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:02.240932 20747 hybrid_clock.cc:648] HybridClock initialized: now 1714685162240886 us; error 36 us; skew 500 ppm
I20240502 21:26:02.241291 20747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:02.243681 20747 webserver.cc:460] Webserver started at http://127.16.122.4:45005/ using document root <none> and password file <none>
I20240502 21:26:02.244544 20747 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:02.244690 20747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:02.245002 20747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:02.247869 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "16745bde14bc4a43b532e6bb828f0625"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:02.248636 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "16745bde14bc4a43b532e6bb828f0625"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:02.253962 20747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240502 21:26:02.257591 20761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.259102 20747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:02.259281 20747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "16745bde14bc4a43b532e6bb828f0625"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:02.259496 20747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:02.277523 20747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:02.278507 20747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:02.278764 20747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:02.279747 20747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:02.281271 20747 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:02.281358 20747 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.281443 20747 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:02.281497 20747 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.314152 20747 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.4:46571
I20240502 21:26:02.314183 20874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.4:46571 every 8 connection(s)
I20240502 21:26:02.315902 20747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:02.318889 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 20747
I20240502 21:26:02.319094 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:26:02.323282 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.5:0
--local_ip_for_outbound_sockets=127.16.122.5
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:02.331884 20875 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:02.332280 20875 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:02.333312 20875 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:02.334903 20287 ts_manager.cc:194] Registered new tserver with Master: 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571)
I20240502 21:26:02.335750 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.4:52483
I20240502 21:26:02.405817 20612 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
W20240502 21:26:02.667378 20878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:02.667742 20878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:02.667894 20878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:02.675500 20878 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:02.675660 20878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:02.675832 20878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.5
I20240502 21:26:02.684974 20878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.5
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:02.686396 20878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:02.688190 20878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:02.701342 20884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:02.702601 20883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:02.703590 20886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:02.704828 20878 server_base.cc:1034] running on GCE node
I20240502 21:26:02.705519 20878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:02.706575 20878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:02.707788 20878 hybrid_clock.cc:648] HybridClock initialized: now 1714685162707741 us; error 49 us; skew 500 ppm
I20240502 21:26:02.708132 20878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:02.716895 20878 webserver.cc:460] Webserver started at http://127.16.122.5:46015/ using document root <none> and password file <none>
I20240502 21:26:02.717772 20878 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:02.717905 20878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:02.718254 20878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:02.723949 20878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ee8dcd502996434ea55a16a62a5324fb"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:02.724812 20878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ee8dcd502996434ea55a16a62a5324fb"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:02.730530 20878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 21:26:02.734082 20892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.735527 20878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:02.735700 20878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ee8dcd502996434ea55a16a62a5324fb"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:02.735914 20878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:02.755616 20878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:02.756539 20878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:02.756822 20878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:02.757830 20878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:02.759531 20878 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:02.759631 20878 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.759756 20878 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:02.759828 20878 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.793224 20878 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.5:35311
I20240502 21:26:02.793285 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.5:35311 every 8 connection(s)
I20240502 21:26:02.794642 20878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:02.803283 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 20878
I20240502 21:26:02.803469 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:26:02.806735 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.6:0
--local_ip_for_outbound_sockets=127.16.122.6
--webserver_interface=127.16.122.6
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:02.807596 21006 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:02.807961 21006 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:02.809073 21006 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:02.810789 20287 ts_manager.cc:194] Registered new tserver with Master: ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311)
I20240502 21:26:02.811576 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.5:39049
I20240502 21:26:02.863741 20744 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
W20240502 21:26:03.154430 21009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:03.154757 21009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:03.154897 21009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:03.162683 21009 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:03.162850 21009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:03.163017 21009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.6
I20240502 21:26:03.172528 21009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.122.6
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.6
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:03.173957 21009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:03.175876 21009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:03.188733 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.189292 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.190474 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:03.191578 21009 server_base.cc:1034] running on GCE node
I20240502 21:26:03.192315 21009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:03.193456 21009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:03.194674 21009 hybrid_clock.cc:648] HybridClock initialized: now 1714685163194627 us; error 42 us; skew 500 ppm
I20240502 21:26:03.194983 21009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:03.197381 21009 webserver.cc:460] Webserver started at http://127.16.122.6:35735/ using document root <none> and password file <none>
I20240502 21:26:03.198146 21009 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:03.198256 21009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:03.198596 21009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:03.201254 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6c0aa2a3f325429eb83423bff3a80761"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:03.202021 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6c0aa2a3f325429eb83423bff3a80761"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:03.207450 21009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:26:03.211017 21023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.212507 21009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:03.212682 21009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c0aa2a3f325429eb83423bff3a80761"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:03.212896 21009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:03.263401 21009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:03.264430 21009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:03.264739 21009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:03.265830 21009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:03.267454 21009 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:03.267556 21009 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.267681 21009 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:03.267755 21009 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.301937 21009 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.6:38855
I20240502 21:26:03.301992 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.6:38855 every 8 connection(s)
I20240502 21:26:03.303470 21009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:03.305248 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 21009
I20240502 21:26:03.305456 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:26:03.309317 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.7:0
--local_ip_for_outbound_sockets=127.16.122.7
--webserver_interface=127.16.122.7
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:03.320310 21137 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:03.320674 21137 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:03.321566 21137 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:03.323910 20287 ts_manager.cc:194] Registered new tserver with Master: 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855)
I20240502 21:26:03.327001 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.6:53035
I20240502 21:26:03.338558 20875 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
W20240502 21:26:03.658515 21141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:03.658830 21141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:03.658946 21141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:03.666687 21141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:03.666816 21141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:03.666972 21141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.7
I20240502 21:26:03.676358 21141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.122.7
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.7
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:03.677775 21141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:03.679597 21141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:03.692276 21146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.692973 21147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.693941 21149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:03.695422 21141 server_base.cc:1034] running on GCE node
I20240502 21:26:03.696074 21141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:03.697090 21141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:03.698315 21141 hybrid_clock.cc:648] HybridClock initialized: now 1714685163698257 us; error 58 us; skew 500 ppm
I20240502 21:26:03.698707 21141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:03.701010 21141 webserver.cc:460] Webserver started at http://127.16.122.7:37291/ using document root <none> and password file <none>
I20240502 21:26:03.701768 21141 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:03.701892 21141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:03.702207 21141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:03.704996 21141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:03.705790 21141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:03.711023 21141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240502 21:26:03.714423 21155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.715775 21141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 21:26:03.715939 21141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:03.716143 21141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:03.748718 21141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:03.749639 21141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:03.749922 21141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:03.750994 21141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:03.752732 21141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:03.752825 21141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.752950 21141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:03.753013 21141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.786216 21141 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.7:34365
I20240502 21:26:03.786289 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.7:34365 every 8 connection(s)
I20240502 21:26:03.787654 21141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:03.792188 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 21141
I20240502 21:26:03.792407 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:26:03.802618 21269 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:03.802913 21269 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:03.803732 21269 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:03.805145 20287 ts_manager.cc:194] Registered new tserver with Master: b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
I20240502 21:26:03.805938 20287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.7:40765
I20240502 21:26:03.809310 16872 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:26:03.814668 21006 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:03.825321 20287 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44234:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:26:03.859511 21203 tablet_service.cc:1467] Processing CreateTablet for tablet 0e14655d22984ef3aab11be93eaaa07d (DEFAULT_TABLE table=TestTable [id=4beabd6b02404fba811a489d4466be1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:03.859784 20409 tablet_service.cc:1467] Processing CreateTablet for tablet 0e14655d22984ef3aab11be93eaaa07d (DEFAULT_TABLE table=TestTable [id=4beabd6b02404fba811a489d4466be1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:03.861130 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e14655d22984ef3aab11be93eaaa07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:03.861146 21203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e14655d22984ef3aab11be93eaaa07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:03.872076 21285 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:03.872893 21286 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap starting.
I20240502 21:26:03.875602 21286 tablet_bootstrap.cc:654] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:03.875603 21285 tablet_bootstrap.cc:654] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:03.876550 21286 log.cc:824] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:03.876549 21285 log.cc:824] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:03.879137 21285 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:03.879154 21286 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: No bootstrap required, opened a new log
I20240502 21:26:03.879413 21286 ts_tablet_manager.cc:1367] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:26:03.879415 21285 ts_tablet_manager.cc:1367] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240502 21:26:03.885304 21286 raft_consensus.cc:348] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:03.885300 21285 raft_consensus.cc:348] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:03.885646 21286 raft_consensus.cc:374] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:03.885664 21285 raft_consensus.cc:374] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:03.885754 21286 raft_consensus.cc:729] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Initialized, Role: FOLLOWER
I20240502 21:26:03.885766 21285 raft_consensus.cc:729] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:03.886447 21285 consensus_queue.cc:260] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:03.886457 21286 consensus_queue.cc:260] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:03.888330 21269 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:03.888480 21285 ts_tablet_manager.cc:1398] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20240502 21:26:03.888872 21286 ts_tablet_manager.cc:1398] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20240502 21:26:03.895959 20476 tablet.cc:2369] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:03.997011 21290 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:03.997368 21290 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:03.999555 21290 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:04.005663 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e14655d22984ef3aab11be93eaaa07d" candidate_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:04.006171 20429 raft_consensus.cc:2454] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4edc0e5b5c04ac5a7fe3f99d86d314b in term 0.
I20240502 21:26:04.006809 21159 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, b4edc0e5b5c04ac5a7fe3f99d86d314b; no voters:
I20240502 21:26:04.007189 21290 raft_consensus.cc:2781] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:04.007349 21290 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:04.007449 21290 raft_consensus.cc:3037] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:04.009585 21290 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:04.010125 21290 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:04.010717 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e14655d22984ef3aab11be93eaaa07d" candidate_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:04.010900 20429 raft_consensus.cc:3037] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:04.013028 20429 raft_consensus.cc:2454] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4edc0e5b5c04ac5a7fe3f99d86d314b in term 1.
I20240502 21:26:04.013433 21159 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, b4edc0e5b5c04ac5a7fe3f99d86d314b; no voters:
I20240502 21:26:04.013727 21290 raft_consensus.cc:2781] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:04.014194 21290 raft_consensus.cc:686] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEADER]: Becoming Leader. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Running, Role: LEADER
I20240502 21:26:04.014776 21290 consensus_queue.cc:237] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:04.020093 20287 catalog_manager.cc:5484] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: term changed from 0 to 1, leader changed from <none> to b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7). New cstate: current_term: 1 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:04.039455 21270 tablet.cc:2369] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:04.047930 16872 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:26:04.049954 16872 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 35812e2216cb4758aee1e94877a823b7 to finish bootstrapping
I20240502 21:26:04.058676 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 654f8d3e16bb4a658607a5c26f261cbf to finish bootstrapping
I20240502 21:26:04.068290 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0ac8c8b961dc4500bc8b9f1055298244 to finish bootstrapping
I20240502 21:26:04.077833 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 16745bde14bc4a43b532e6bb828f0625 to finish bootstrapping
I20240502 21:26:04.087934 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ee8dcd502996434ea55a16a62a5324fb to finish bootstrapping
I20240502 21:26:04.097594 16872 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6c0aa2a3f325429eb83423bff3a80761 to finish bootstrapping
I20240502 21:26:04.106492 16872 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b4edc0e5b5c04ac5a7fe3f99d86d314b to finish bootstrapping
I20240502 21:26:04.114267 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 20747
I20240502 21:26:04.115537 20870 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:04.198257 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 20747
I20240502 21:26:04.216547 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 20878
I20240502 21:26:04.217764 21001 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:04.300331 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 20878
I20240502 21:26:04.322490 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 21009
I20240502 21:26:04.323633 21132 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:04.330041 21137 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:04.413511 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 21009
I20240502 21:26:04.430269 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 21141
I20240502 21:26:04.431499 21264 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:04.520586 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 21141
I20240502 21:26:05.812386 21303 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:05.812623 21303 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:05.814252 21303 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
W20240502 21:26:05.815543 20362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:34365: connect: Connection refused (error 111)
W20240502 21:26:05.819989 20362 leader_election.cc:334] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Network error: Client connection negotiation failed: client connection to 127.16.122.7:34365: connect: Connection refused (error 111)
I20240502 21:26:05.820186 20362 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7; no voters: b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:05.820689 21303 raft_consensus.cc:2726] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:07.540521 21306 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:07.540720 21306 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:07.541304 21306 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
W20240502 21:26:07.543591 20362 leader_election.cc:334] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Network error: Client connection negotiation failed: client connection to 127.16.122.7:34365: connect: Connection refused (error 111)
I20240502 21:26:07.543736 20362 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7; no voters: b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:07.544059 21306 raft_consensus.cc:2726] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:08.290431 20287 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44234:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:08.304116 20409 tablet_service.cc:1467] Processing CreateTablet for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:08.304663 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.319416 21317 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:08.323455 21317 tablet_bootstrap.cc:654] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.325949 21317 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:08.326138 21317 ts_tablet_manager.cc:1367] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:08.327623 21317 raft_consensus.cc:348] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.327936 21317 raft_consensus.cc:374] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.328095 21317 raft_consensus.cc:729] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.329279 21317 consensus_queue.cc:260] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.328505 20679 tablet_service.cc:1467] Processing CreateTablet for tablet 44116b77039245b4883892cf19a8a248 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:08.330746 21317 ts_tablet_manager.cc:1398] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:26:08.328440 20678 tablet_service.cc:1467] Processing CreateTablet for tablet d15373bff59348b9a65b42053e8df389 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:08.331606 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15373bff59348b9a65b42053e8df389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.332459 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44116b77039245b4883892cf19a8a248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.332341 20543 tablet_service.cc:1467] Processing CreateTablet for tablet d15373bff59348b9a65b42053e8df389 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:08.333339 20544 tablet_service.cc:1467] Processing CreateTablet for tablet 44116b77039245b4883892cf19a8a248 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:08.334069 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15373bff59348b9a65b42053e8df389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.334607 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44116b77039245b4883892cf19a8a248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.334044 20542 tablet_service.cc:1467] Processing CreateTablet for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:08.335110 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.348732 21321 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:08.352025 21321 tablet_bootstrap.cc:654] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.353365 21321 log.cc:824] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:08.354809 21320 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:08.356654 21321 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:08.356946 21321 ts_tablet_manager.cc:1367] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:26:08.358649 21320 tablet_bootstrap.cc:654] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.359980 21320 log.cc:824] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:08.363241 21320 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:08.363557 21320 ts_tablet_manager.cc:1367] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 21:26:08.366045 21321 raft_consensus.cc:348] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.366561 21321 raft_consensus.cc:374] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.366674 21321 raft_consensus.cc:729] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.367472 21321 consensus_queue.cc:260] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.369323 21321 ts_tablet_manager.cc:1398] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20240502 21:26:08.369809 21321 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:08.371629 21321 tablet_bootstrap.cc:654] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.372716 21320 raft_consensus.cc:348] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.373281 21320 raft_consensus.cc:374] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.373370 21320 raft_consensus.cc:729] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.374034 21320 consensus_queue.cc:260] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.374486 21321 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:08.374666 21321 ts_tablet_manager.cc:1367] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:08.375600 21321 raft_consensus.cc:348] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.375806 21321 raft_consensus.cc:374] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.375913 21321 raft_consensus.cc:729] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.375952 21320 ts_tablet_manager.cc:1398] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20240502 21:26:08.376197 21321 consensus_queue.cc:260] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.376468 21320 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:08.376695 21321 ts_tablet_manager.cc:1398] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:08.376964 21321 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:08.378260 21320 tablet_bootstrap.cc:654] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.378751 21321 tablet_bootstrap.cc:654] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:08.383404 21321 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:08.383637 21321 ts_tablet_manager.cc:1367] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:08.384351 21321 raft_consensus.cc:348] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.384578 21321 raft_consensus.cc:374] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.384670 21321 raft_consensus.cc:729] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.385048 21320 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:08.384948 21321 consensus_queue.cc:260] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.385216 21320 ts_tablet_manager.cc:1367] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:26:08.385556 21321 ts_tablet_manager.cc:1398] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:08.386464 21320 raft_consensus.cc:348] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.386704 21320 raft_consensus.cc:374] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:08.386821 21320 raft_consensus.cc:729] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:08.387148 21320 consensus_queue.cc:260] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.387816 21320 ts_tablet_manager.cc:1398] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240502 21:26:08.390264 20614 tablet.cc:2369] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:08.414712 21324 raft_consensus.cc:482] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:08.415040 21324 raft_consensus.cc:504] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.416769 21324 leader_election.cc:288] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:08.424084 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a0d6a8a5a348dba1e8fd42d46b5f61" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:08.424350 20429 raft_consensus.cc:2454] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:08.424903 20497 leader_election.cc:302] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:08.425323 21324 raft_consensus.cc:2781] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:08.425477 21324 raft_consensus.cc:482] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:08.425580 21324 raft_consensus.cc:3037] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.427717 21324 raft_consensus.cc:504] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.428238 21324 leader_election.cc:288] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:08.428836 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a0d6a8a5a348dba1e8fd42d46b5f61" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:08.429049 20429 raft_consensus.cc:3037] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.431125 20429 raft_consensus.cc:2454] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:08.431576 20497 leader_election.cc:302] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:08.431882 21324 raft_consensus.cc:2781] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:08.432343 21324 raft_consensus.cc:686] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:08.432896 21324 consensus_queue.cc:237] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.437794 20287 catalog_manager.cc:5484] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } }
W20240502 21:26:08.601125 20745 tablet.cc:2369] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:08.691538 21326 raft_consensus.cc:482] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:08.691846 21326 raft_consensus.cc:504] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.693593 21326 leader_election.cc:288] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:08.695210 21326 raft_consensus.cc:482] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:08.695434 21326 raft_consensus.cc:504] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.696034 21326 leader_election.cc:288] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:08.700569 20565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15373bff59348b9a65b42053e8df389" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf" is_pre_election: true
I20240502 21:26:08.700595 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44116b77039245b4883892cf19a8a248" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf" is_pre_election: true
I20240502 21:26:08.701066 20564 raft_consensus.cc:2454] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 0.
I20240502 21:26:08.701062 20565 raft_consensus.cc:2454] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 0.
I20240502 21:26:08.701748 20632 leader_election.cc:302] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:08.702189 21326 raft_consensus.cc:2781] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:08.702314 21326 raft_consensus.cc:482] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:08.702432 20632 leader_election.cc:302] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:08.702522 21326 raft_consensus.cc:3037] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.703210 21334 raft_consensus.cc:2781] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:08.703310 21334 raft_consensus.cc:482] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:08.703440 21334 raft_consensus.cc:3037] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.705075 21326 raft_consensus.cc:504] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.705580 21326 leader_election.cc:288] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:08.705519 21334 raft_consensus.cc:504] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.705991 21334 leader_election.cc:288] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:08.706152 20565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15373bff59348b9a65b42053e8df389" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:08.706411 20565 raft_consensus.cc:3037] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.706549 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44116b77039245b4883892cf19a8a248" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:08.706737 20564 raft_consensus.cc:3037] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:08.708762 20565 raft_consensus.cc:2454] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 1.
I20240502 21:26:08.708873 20564 raft_consensus.cc:2454] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 1.
I20240502 21:26:08.709303 20632 leader_election.cc:302] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:08.709611 21334 raft_consensus.cc:2781] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:08.709754 20632 leader_election.cc:302] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:08.709811 21334 raft_consensus.cc:686] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:08.710018 21326 raft_consensus.cc:2781] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:08.710304 21334 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:08.710608 21326 raft_consensus.cc:686] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:08.710883 21326 consensus_queue.cc:237] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:08.715406 20287 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:08.720048 20287 catalog_manager.cc:5484] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:08.863883 21324 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:09.091588 21334 consensus_queue.cc:1035] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:09.120126 21334 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:09.147095 21338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:09.147486 21338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:09.155390 21338 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 192.122 ms
time per row: 0.0960612 ms
I20240502 21:26:09.495752 20282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44234:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:09.504885 20409 tablet_service.cc:1467] Processing CreateTablet for tablet 2998322d830948ac858235f4fb7defca (DEFAULT_TABLE table=rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:09.505409 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998322d830948ac858235f4fb7defca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:09.505299 20679 tablet_service.cc:1467] Processing CreateTablet for tablet 2998322d830948ac858235f4fb7defca (DEFAULT_TABLE table=rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:09.505371 20408 tablet_service.cc:1467] Processing CreateTablet for tablet 028bf9972bf44bbbad3b862c70b174ca (DEFAULT_TABLE table=rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:09.505754 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998322d830948ac858235f4fb7defca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:09.505851 20408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028bf9972bf44bbbad3b862c70b174ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:09.506116 20544 tablet_service.cc:1467] Processing CreateTablet for tablet 7875028620db4a8bbad7f0c5d1331f5e (DEFAULT_TABLE table=rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:09.506593 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7875028620db4a8bbad7f0c5d1331f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:09.511114 20407 tablet_service.cc:1467] Processing CreateTablet for tablet 7875028620db4a8bbad7f0c5d1331f5e (DEFAULT_TABLE table=rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:09.511703 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7875028620db4a8bbad7f0c5d1331f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:09.511685 20678 tablet_service.cc:1467] Processing CreateTablet for tablet 028bf9972bf44bbbad3b862c70b174ca (DEFAULT_TABLE table=rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:09.512117 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028bf9972bf44bbbad3b862c70b174ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:09.520676 21395 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:09.520774 21396 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:09.521564 21394 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:09.522665 21395 tablet_bootstrap.cc:654] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:09.523119 21396 tablet_bootstrap.cc:654] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:09.523797 21394 tablet_bootstrap.cc:654] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:09.529858 21394 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:09.530261 21394 ts_tablet_manager.cc:1367] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:26:09.530335 21400 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:09.531558 21394 raft_consensus.cc:348] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.531507 21400 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:09.531889 21394 raft_consensus.cc:374] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:09.532063 21394 raft_consensus.cc:729] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:09.532313 21400 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
I20240502 21:26:09.533097 21394 consensus_queue.cc:260] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.534976 21394 ts_tablet_manager.cc:1398] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:09.535276 21394 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:09.535547 21395 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:09.537034 21394 tablet_bootstrap.cc:654] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:09.535696 21395 ts_tablet_manager.cc:1367] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240502 21:26:09.538187 21395 raft_consensus.cc:348] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:09.538501 21395 raft_consensus.cc:374] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:09.538885 21396 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:09.539073 21396 ts_tablet_manager.cc:1367] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.006s
I20240502 21:26:09.539994 21396 raft_consensus.cc:348] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:09.540217 21396 raft_consensus.cc:374] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:09.540338 21396 raft_consensus.cc:729] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:09.540622 21396 consensus_queue.cc:260] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
W20240502 21:26:09.540886 20362 leader_election.cc:334] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Network error: Client connection negotiation failed: client connection to 127.16.122.7:34365: connect: Connection refused (error 111)
I20240502 21:26:09.541081 21395 raft_consensus.cc:729] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:09.541343 20362 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7; no voters: b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:09.541417 21396 ts_tablet_manager.cc:1398] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:09.541374 21395 consensus_queue.cc:260] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:09.542044 21395 ts_tablet_manager.cc:1398] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:26:09.542394 21400 raft_consensus.cc:2726] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:09.542429 21396 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:09.542930 21394 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:09.543097 21394 ts_tablet_manager.cc:1367] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 21:26:09.544027 21394 raft_consensus.cc:348] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.544209 21396 tablet_bootstrap.cc:654] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:09.544294 21394 raft_consensus.cc:374] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:09.545053 21394 raft_consensus.cc:729] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:09.545377 21394 consensus_queue.cc:260] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.545820 21326 raft_consensus.cc:482] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:09.545951 21326 raft_consensus.cc:504] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.546710 21396 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:09.546707 21326 leader_election.cc:288] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:09.546882 21396 ts_tablet_manager.cc:1367] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:09.547724 21396 raft_consensus.cc:348] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.547960 21396 raft_consensus.cc:374] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:09.548056 21396 raft_consensus.cc:729] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:09.548296 21396 consensus_queue.cc:260] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.548925 21396 ts_tablet_manager.cc:1398] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:09.549224 21396 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:09.550969 21396 tablet_bootstrap.cc:654] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:09.552223 21394 ts_tablet_manager.cc:1398] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 21:26:09.556599 21396 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:09.556775 21396 ts_tablet_manager.cc:1367] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:26:09.557507 21396 raft_consensus.cc:348] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.557739 21396 raft_consensus.cc:374] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:09.557864 21396 raft_consensus.cc:729] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:09.558328 21396 consensus_queue.cc:260] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.558909 21396 ts_tablet_manager.cc:1398] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:09.562633 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998322d830948ac858235f4fb7defca" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:09.562856 20429 raft_consensus.cc:2454] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 0.
I20240502 21:26:09.563377 20635 leader_election.cc:302] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:09.563741 21386 raft_consensus.cc:2781] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:09.563839 21386 raft_consensus.cc:482] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:09.563972 21386 raft_consensus.cc:3037] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:09.566098 21386 raft_consensus.cc:504] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.566663 21386 leader_election.cc:288] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:09.567200 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998322d830948ac858235f4fb7defca" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:09.567399 20429 raft_consensus.cc:3037] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:09.569365 20429 raft_consensus.cc:2454] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 1.
I20240502 21:26:09.569825 20635 leader_election.cc:302] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:09.570132 21386 raft_consensus.cc:2781] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:09.570283 21386 raft_consensus.cc:686] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:09.570565 21386 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.573273 20286 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:09.613880 21386 raft_consensus.cc:482] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:09.614089 21386 raft_consensus.cc:504] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.614761 21386 leader_election.cc:288] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:09.615402 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028bf9972bf44bbbad3b862c70b174ca" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:09.615633 20429 raft_consensus.cc:2454] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 0.
I20240502 21:26:09.616045 20635 leader_election.cc:302] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:09.616336 21386 raft_consensus.cc:2781] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:09.616467 21386 raft_consensus.cc:482] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:09.616541 21386 raft_consensus.cc:3037] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:09.618569 21386 raft_consensus.cc:504] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.619042 21386 leader_election.cc:288] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:09.619655 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028bf9972bf44bbbad3b862c70b174ca" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:09.619854 20429 raft_consensus.cc:3037] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:09.621877 20429 raft_consensus.cc:2454] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 1.
I20240502 21:26:09.622336 20635 leader_election.cc:302] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:09.622732 21386 raft_consensus.cc:2781] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:09.622881 21386 raft_consensus.cc:686] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:09.623144 21386 consensus_queue.cc:237] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:09.625963 20286 catalog_manager.cc:5484] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:09.835412 21400 raft_consensus.cc:482] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:09.835589 21400 raft_consensus.cc:504] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:09.836223 21400 leader_election.cc:288] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:09.842021 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf" is_pre_election: true
I20240502 21:26:09.842257 20564 raft_consensus.cc:2454] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 0.
I20240502 21:26:09.842774 20363 leader_election.cc:302] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:09.843122 21400 raft_consensus.cc:2781] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:09.843273 21400 raft_consensus.cc:482] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:09.843379 21400 raft_consensus.cc:3037] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:09.845501 21400 raft_consensus.cc:504] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:09.845957 21400 leader_election.cc:288] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:09.846520 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:09.846731 20564 raft_consensus.cc:3037] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:09.848542 20564 raft_consensus.cc:2454] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 1.
I20240502 21:26:09.848902 20363 leader_election.cc:302] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:09.849176 21400 raft_consensus.cc:2781] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:09.849634 21400 raft_consensus.cc:686] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Becoming Leader. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Running, Role: LEADER
I20240502 21:26:09.850159 21400 consensus_queue.cc:237] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:09.853801 20286 catalog_manager.cc:5484] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35812e2216cb4758aee1e94877a823b7 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:10.120647 21386 consensus_queue.cc:1035] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:10.139840 21415 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:10.253204 21413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:10.253598 21413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:10.262506 21413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 21:26:10.397868 20564 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:10.398768 21400 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 194.272 ms
time per row: 0.097136 ms
I20240502 21:26:10.606465 20283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44234:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:10.616832 20678 tablet_service.cc:1467] Processing CreateTablet for tablet 3fabe357cd7d42259e10511baa31b0bc (DEFAULT_TABLE table=rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.617554 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fabe357cd7d42259e10511baa31b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.617733 20679 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca7cd6b72d4419aade387eaf375454e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.618144 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca7cd6b72d4419aade387eaf375454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.618207 20409 tablet_service.cc:1467] Processing CreateTablet for tablet d0892742cff44146ae2fb345d7e2bd3a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.618320 20544 tablet_service.cc:1467] Processing CreateTablet for tablet 3fabe357cd7d42259e10511baa31b0bc (DEFAULT_TABLE table=rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.618755 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fabe357cd7d42259e10511baa31b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.618763 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0892742cff44146ae2fb345d7e2bd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.622761 20542 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca7cd6b72d4419aade387eaf375454e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.623147 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca7cd6b72d4419aade387eaf375454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.625734 21472 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:10.626762 20543 tablet_service.cc:1467] Processing CreateTablet for tablet d0892742cff44146ae2fb345d7e2bd3a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.627148 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0892742cff44146ae2fb345d7e2bd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.627571 21472 tablet_bootstrap.cc:654] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.630131 21472 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:10.630323 21472 ts_tablet_manager.cc:1367] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:10.631182 21472 raft_consensus.cc:348] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.631417 21472 raft_consensus.cc:374] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.631542 21472 raft_consensus.cc:729] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.631799 21472 consensus_queue.cc:260] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.632467 21472 ts_tablet_manager.cc:1398] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:10.632817 21472 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:10.634459 21472 tablet_bootstrap.cc:654] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.634766 21473 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:10.637673 21473 tablet_bootstrap.cc:654] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.641714 21477 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:10.642834 21472 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:10.643093 21472 ts_tablet_manager.cc:1367] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 21:26:10.643435 21477 tablet_bootstrap.cc:654] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.643498 21473 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:10.643711 21473 ts_tablet_manager.cc:1367] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240502 21:26:10.644196 21472 raft_consensus.cc:348] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.644495 21472 raft_consensus.cc:374] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.644604 21472 raft_consensus.cc:729] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.644614 21473 raft_consensus.cc:348] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.644894 21473 raft_consensus.cc:374] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.645068 21473 raft_consensus.cc:729] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.645288 21477 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:10.645423 21477 ts_tablet_manager.cc:1367] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:10.645332 21473 consensus_queue.cc:260] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.646030 21473 ts_tablet_manager.cc:1398] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:10.646416 21472 consensus_queue.cc:260] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.647001 21472 ts_tablet_manager.cc:1398] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:10.646909 21477 raft_consensus.cc:348] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.647107 21477 raft_consensus.cc:374] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.647258 21477 raft_consensus.cc:729] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.647524 21477 consensus_queue.cc:260] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.648700 21477 ts_tablet_manager.cc:1398] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:10.649008 21477 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:10.650614 21477 tablet_bootstrap.cc:654] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.658896 21477 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:10.659098 21477 ts_tablet_manager.cc:1367] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 21:26:10.659924 21477 raft_consensus.cc:348] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.660115 21477 raft_consensus.cc:374] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.660193 21477 raft_consensus.cc:729] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.660439 21477 consensus_queue.cc:260] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.660914 21477 ts_tablet_manager.cc:1398] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:10.661199 21477 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:10.662616 21477 tablet_bootstrap.cc:654] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.664554 21477 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:10.664682 21477 ts_tablet_manager.cc:1367] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:10.665462 21477 raft_consensus.cc:348] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.665628 21477 raft_consensus.cc:374] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.665697 21477 raft_consensus.cc:729] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.665906 21477 consensus_queue.cc:260] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.666402 21477 ts_tablet_manager.cc:1398] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:10.758142 21450 raft_consensus.cc:482] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:10.758432 21450 raft_consensus.cc:504] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.759145 21450 leader_election.cc:288] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:10.759733 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0892742cff44146ae2fb345d7e2bd3a" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:10.759935 20429 raft_consensus.cc:2454] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:10.760468 20497 leader_election.cc:302] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:10.760792 21450 raft_consensus.cc:2781] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:10.760932 21450 raft_consensus.cc:482] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:10.761023 21450 raft_consensus.cc:3037] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.763128 21450 raft_consensus.cc:504] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.763659 21450 leader_election.cc:288] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:10.764252 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0892742cff44146ae2fb345d7e2bd3a" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:10.764463 20429 raft_consensus.cc:3037] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.766427 20429 raft_consensus.cc:2454] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:10.766891 20497 leader_election.cc:302] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:10.767215 21450 raft_consensus.cc:2781] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:10.767376 21450 raft_consensus.cc:686] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:10.767645 21450 consensus_queue.cc:237] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:10.770275 20286 catalog_manager.cc:5484] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:10.776654 21450 raft_consensus.cc:482] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:10.776836 21450 raft_consensus.cc:504] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.777412 21450 leader_election.cc:288] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:10.783826 20699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fabe357cd7d42259e10511baa31b0bc" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244" is_pre_election: true
I20240502 21:26:10.784237 20699 raft_consensus.cc:2454] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:10.784708 20496 leader_election.cc:302] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:10.784998 21450 raft_consensus.cc:2781] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:10.785085 21450 raft_consensus.cc:482] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:10.785197 21450 raft_consensus.cc:3037] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.787187 21450 raft_consensus.cc:504] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.787691 21450 leader_election.cc:288] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:10.788187 20699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fabe357cd7d42259e10511baa31b0bc" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:10.788398 20699 raft_consensus.cc:3037] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.790766 20699 raft_consensus.cc:2454] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:10.791127 20496 leader_election.cc:302] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:10.791431 21450 raft_consensus.cc:2781] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:10.791596 21450 raft_consensus.cc:686] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:10.791819 21450 consensus_queue.cc:237] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:10.794425 20286 catalog_manager.cc:5484] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:10.883086 21450 raft_consensus.cc:482] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:10.883313 21450 raft_consensus.cc:504] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.883875 21450 leader_election.cc:288] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:10.884441 20699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244" is_pre_election: true
I20240502 21:26:10.884692 20699 raft_consensus.cc:2454] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:10.885128 20496 leader_election.cc:302] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:10.885448 21449 raft_consensus.cc:2781] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:10.885558 21449 raft_consensus.cc:482] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:10.885663 21449 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.887934 21449 raft_consensus.cc:504] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.888407 21449 leader_election.cc:288] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:10.888976 20699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:10.889199 20699 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.891789 20699 raft_consensus.cc:2454] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:10.892333 20496 leader_election.cc:302] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:10.892675 21449 raft_consensus.cc:2781] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:10.892858 21449 raft_consensus.cc:686] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:10.893155 21449 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:10.895932 20286 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:11.278079 21485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:11.278558 21485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:11.286444 21485 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:11.312505 21449 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 21:26:11.325493 21449 consensus_queue.cc:1035] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:11.388708 21449 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 159.888 ms
time per row: 0.0799441 ms
I20240502 21:26:11.601318 20283 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44234:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:11.610921 20678 tablet_service.cc:1467] Processing CreateTablet for tablet 12887227115743afacb541c753db51c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:11.611112 20409 tablet_service.cc:1467] Processing CreateTablet for tablet 12887227115743afacb541c753db51c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:11.611474 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12887227115743afacb541c753db51c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.611543 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12887227115743afacb541c753db51c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.611424 20408 tablet_service.cc:1467] Processing CreateTablet for tablet ef774f42c72349d0aa9d490bb5dd8011 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:11.611876 20543 tablet_service.cc:1467] Processing CreateTablet for tablet 129cc6d483b842fead09a5b1ddd58f52 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:11.612025 20408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef774f42c72349d0aa9d490bb5dd8011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.612334 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129cc6d483b842fead09a5b1ddd58f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.619422 20679 tablet_service.cc:1467] Processing CreateTablet for tablet ef774f42c72349d0aa9d490bb5dd8011 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:11.619925 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef774f42c72349d0aa9d490bb5dd8011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.620517 20408 tablet_service.cc:1467] Processing CreateTablet for tablet 129cc6d483b842fead09a5b1ddd58f52 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:11.620905 20408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129cc6d483b842fead09a5b1ddd58f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:11.622071 21538 tablet_bootstrap.cc:492] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:11.622758 21536 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:11.624007 21538 tablet_bootstrap.cc:654] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:11.624617 21536 tablet_bootstrap.cc:654] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:11.629488 21537 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:11.631891 21537 tablet_bootstrap.cc:654] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:11.633617 21538 tablet_bootstrap.cc:492] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:11.633613 21536 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:11.633781 21536 ts_tablet_manager.cc:1367] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 21:26:11.633785 21538 ts_tablet_manager.cc:1367] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240502 21:26:11.634079 21537 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:11.634236 21537 ts_tablet_manager.cc:1367] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 21:26:11.634579 21538 raft_consensus.cc:348] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.634670 21536 raft_consensus.cc:348] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.634872 21538 raft_consensus.cc:374] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:11.634954 21536 raft_consensus.cc:374] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:11.635082 21538 raft_consensus.cc:729] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:11.635105 21536 raft_consensus.cc:729] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:11.635033 21537 raft_consensus.cc:348] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.635347 21537 raft_consensus.cc:374] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:11.635353 21536 consensus_queue.cc:260] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.635488 21537 raft_consensus.cc:729] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:11.635409 21538 consensus_queue.cc:260] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.635826 21537 consensus_queue.cc:260] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.636022 21538 ts_tablet_manager.cc:1398] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:11.637035 21537 ts_tablet_manager.cc:1398] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:11.637391 21537 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:11.638046 21536 ts_tablet_manager.cc:1398] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:11.638402 21536 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:11.639391 21537 tablet_bootstrap.cc:654] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:11.640305 21536 tablet_bootstrap.cc:654] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:11.642619 21537 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:11.642822 21537 ts_tablet_manager.cc:1367] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 21:26:11.643347 21536 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:11.643498 21536 ts_tablet_manager.cc:1367] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:11.643772 21537 raft_consensus.cc:348] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.644006 21537 raft_consensus.cc:374] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:11.644070 21537 raft_consensus.cc:729] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:11.644124 21536 raft_consensus.cc:348] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.644346 21536 raft_consensus.cc:374] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:11.644409 21536 raft_consensus.cc:729] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:11.644376 21537 consensus_queue.cc:260] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.644610 21536 consensus_queue.cc:260] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.644995 21537 ts_tablet_manager.cc:1398] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:11.645079 21536 ts_tablet_manager.cc:1398] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:11.645323 21537 tablet_bootstrap.cc:492] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:11.647280 21537 tablet_bootstrap.cc:654] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:11.649618 21455 raft_consensus.cc:482] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:11.649760 21455 raft_consensus.cc:504] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.650274 21455 leader_election.cc:288] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:11.650523 21400 raft_consensus.cc:482] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:11.650684 21400 raft_consensus.cc:504] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.651033 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef774f42c72349d0aa9d490bb5dd8011" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:11.651288 21400 leader_election.cc:288] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:11.651365 20429 raft_consensus.cc:2454] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 0.
I20240502 21:26:11.652501 20635 leader_election.cc:302] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:11.652911 21455 raft_consensus.cc:2781] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:11.653030 21455 raft_consensus.cc:482] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:11.653123 21455 raft_consensus.cc:3037] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:11.656365 21455 raft_consensus.cc:504] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.656877 21455 leader_election.cc:288] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:11.657605 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef774f42c72349d0aa9d490bb5dd8011" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:11.657888 20429 raft_consensus.cc:3037] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:11.661031 20429 raft_consensus.cc:2454] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 1.
I20240502 21:26:11.661484 20635 leader_election.cc:302] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:11.662003 21455 raft_consensus.cc:2781] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:11.662024 21537 tablet_bootstrap.cc:492] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:11.662206 21537 ts_tablet_manager.cc:1367] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20240502 21:26:11.662448 21455 raft_consensus.cc:686] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:11.662817 21455 consensus_queue.cc:237] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:11.662973 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef774f42c72349d0aa9d490bb5dd8011" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244" is_pre_election: true
I20240502 21:26:11.663213 21537 raft_consensus.cc:348] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.663365 21537 raft_consensus.cc:374] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:11.663522 21537 raft_consensus.cc:729] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:11.663662 20364 leader_election.cc:302] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7; no voters: 0ac8c8b961dc4500bc8b9f1055298244
I20240502 21:26:11.663782 21537 consensus_queue.cc:260] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.664038 21400 raft_consensus.cc:2726] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:11.667917 21537 ts_tablet_manager.cc:1398] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240502 21:26:11.670225 20283 catalog_manager.cc:5484] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:11.703513 21549 raft_consensus.cc:482] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:11.703749 21549 raft_consensus.cc:504] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.704391 21549 leader_election.cc:288] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:11.705062 20565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129cc6d483b842fead09a5b1ddd58f52" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf" is_pre_election: true
I20240502 21:26:11.705322 20565 raft_consensus.cc:2454] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 0.
I20240502 21:26:11.705780 20363 leader_election.cc:302] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:11.706065 21549 raft_consensus.cc:2781] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:11.706176 21549 raft_consensus.cc:482] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:11.706267 21549 raft_consensus.cc:3037] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:11.708330 21549 raft_consensus.cc:504] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.708743 21549 leader_election.cc:288] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:11.709203 20565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129cc6d483b842fead09a5b1ddd58f52" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:11.709394 20565 raft_consensus.cc:3037] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:11.711359 20565 raft_consensus.cc:2454] T 129cc6d483b842fead09a5b1ddd58f52 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 1.
I20240502 21:26:11.711757 20363 leader_election.cc:302] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:11.712047 21549 raft_consensus.cc:2781] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:11.712215 21549 raft_consensus.cc:686] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Becoming Leader. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Running, Role: LEADER
I20240502 21:26:11.712507 21549 consensus_queue.cc:237] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.715318 20283 catalog_manager.cc:5484] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35812e2216cb4758aee1e94877a823b7 (127.16.122.1). New cstate: current_term: 1 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:11.839555 21455 raft_consensus.cc:482] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:11.839777 21455 raft_consensus.cc:504] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.840312 21455 leader_election.cc:288] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:11.840983 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12887227115743afacb541c753db51c9" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:11.841199 20429 raft_consensus.cc:2454] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 0.
I20240502 21:26:11.841662 20635 leader_election.cc:302] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:11.841946 21455 raft_consensus.cc:2781] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:11.842038 21455 raft_consensus.cc:482] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:11.842123 21455 raft_consensus.cc:3037] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:11.844193 21455 raft_consensus.cc:504] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.844702 21455 leader_election.cc:288] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:11.845319 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12887227115743afacb541c753db51c9" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:11.845535 20429 raft_consensus.cc:3037] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:11.847599 20429 raft_consensus.cc:2454] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 1.
I20240502 21:26:11.848054 20635 leader_election.cc:302] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:11.848351 21455 raft_consensus.cc:2781] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:11.848524 21455 raft_consensus.cc:686] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:11.848785 21455 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:11.851737 20283 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:12.113415 21549 consensus_queue.cc:1035] T 129cc6d483b842fead09a5b1ddd58f52 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:12.180966 21455 consensus_queue.cc:1035] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:12.224820 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:12.225162 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:12.229488 21547 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:12.234622 21552 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 206.438 ms
time per row: 0.103219 ms
I20240502 21:26:12.588876 20287 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44234:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:12.598239 20543 tablet_service.cc:1467] Processing CreateTablet for tablet d4aeebd7828643ccab4aa62d4fd4d12a (DEFAULT_TABLE table=rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.598825 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aeebd7828643ccab4aa62d4fd4d12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.598788 20679 tablet_service.cc:1467] Processing CreateTablet for tablet d4aeebd7828643ccab4aa62d4fd4d12a (DEFAULT_TABLE table=rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.598866 20544 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f (DEFAULT_TABLE table=rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.599228 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.599225 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aeebd7828643ccab4aa62d4fd4d12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.599443 20542 tablet_service.cc:1467] Processing CreateTablet for tablet 380fd60c78334d90b26c3c9ce4945247 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.599838 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380fd60c78334d90b26c3c9ce4945247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.603871 20678 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f (DEFAULT_TABLE table=rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.604439 20408 tablet_service.cc:1467] Processing CreateTablet for tablet 380fd60c78334d90b26c3c9ce4945247 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.604899 20408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380fd60c78334d90b26c3c9ce4945247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.606231 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.617384 21607 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:12.618021 21605 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:12.619530 21607 tablet_bootstrap.cc:654] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.620797 21605 tablet_bootstrap.cc:654] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.622109 21606 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7: Bootstrap starting.
I20240502 21:26:12.622279 21607 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:12.622452 21607 ts_tablet_manager.cc:1367] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:26:12.623239 21607 raft_consensus.cc:348] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.623478 21605 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:12.623481 21607 raft_consensus.cc:374] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.623587 21607 raft_consensus.cc:729] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.623648 21605 ts_tablet_manager.cc:1367] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:26:12.623865 21607 consensus_queue.cc:260] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.624006 21606 tablet_bootstrap.cc:654] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.624388 21605 raft_consensus.cc:348] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.624634 21605 raft_consensus.cc:374] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.624758 21605 raft_consensus.cc:729] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.625018 21605 consensus_queue.cc:260] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.625458 21607 ts_tablet_manager.cc:1398] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:12.625595 21605 ts_tablet_manager.cc:1398] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:12.625770 21607 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: Bootstrap starting.
I20240502 21:26:12.626557 21605 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:12.629112 21607 tablet_bootstrap.cc:654] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.629151 21605 tablet_bootstrap.cc:654] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.631934 21606 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7: No bootstrap required, opened a new log
I20240502 21:26:12.632161 21606 ts_tablet_manager.cc:1367] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:26:12.632985 21605 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:12.633051 21607 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: No bootstrap required, opened a new log
I20240502 21:26:12.633145 21605 ts_tablet_manager.cc:1367] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240502 21:26:12.633054 21606 raft_consensus.cc:348] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.633219 21607 ts_tablet_manager.cc:1367] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 21:26:12.633286 21606 raft_consensus.cc:374] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.633383 21606 raft_consensus.cc:729] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.633648 21606 consensus_queue.cc:260] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.633766 21605 raft_consensus.cc:348] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.633944 21605 raft_consensus.cc:374] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.634017 21605 raft_consensus.cc:729] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.634008 21607 raft_consensus.cc:348] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.634229 21607 raft_consensus.cc:374] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.634269 21606 ts_tablet_manager.cc:1398] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:12.634305 21607 raft_consensus.cc:729] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.634279 21605 consensus_queue.cc:260] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.634603 21607 consensus_queue.cc:260] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.634896 21605 ts_tablet_manager.cc:1398] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:12.635111 21607 ts_tablet_manager.cc:1398] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:12.635329 21605 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Bootstrap starting.
I20240502 21:26:12.637037 21605 tablet_bootstrap.cc:654] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.639122 21605 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: No bootstrap required, opened a new log
I20240502 21:26:12.639268 21605 ts_tablet_manager.cc:1367] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:26:12.639855 21605 raft_consensus.cc:348] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.640074 21605 raft_consensus.cc:374] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.640183 21605 raft_consensus.cc:729] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.640465 21605 consensus_queue.cc:260] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.640890 21605 ts_tablet_manager.cc:1398] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:12.652882 21549 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.653131 21549 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:12.653818 21549 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
W20240502 21:26:12.654894 20362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:34365: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240502 21:26:12.655254 21553 raft_consensus.cc:482] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.655468 21553 raft_consensus.cc:504] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.656118 21553 leader_election.cc:288] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
W20240502 21:26:12.656651 20362 leader_election.cc:334] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Network error: Client connection negotiation failed: client connection to 127.16.122.7:34365: connect: Connection refused (error 111)
I20240502 21:26:12.656792 20362 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7; no voters: b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:12.656852 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdf4a8993904b02b2dd0fe3ba0f9e4f" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244" is_pre_election: true
I20240502 21:26:12.657090 21549 raft_consensus.cc:2726] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:26:12.657110 20698 raft_consensus.cc:2454] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:12.657586 20496 leader_election.cc:302] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:12.657893 21553 raft_consensus.cc:2781] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:12.658015 21553 raft_consensus.cc:482] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:12.658102 21553 raft_consensus.cc:3037] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.660145 21553 raft_consensus.cc:504] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.660646 21553 leader_election.cc:288] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:12.661228 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdf4a8993904b02b2dd0fe3ba0f9e4f" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:12.661443 20698 raft_consensus.cc:3037] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.663578 20698 raft_consensus.cc:2454] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:12.664389 20496 leader_election.cc:302] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:12.664733 21553 raft_consensus.cc:2781] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:12.665174 21553 raft_consensus.cc:686] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:12.665472 21553 consensus_queue.cc:237] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.668155 20287 catalog_manager.cc:5484] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:12.739244 21553 raft_consensus.cc:482] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.739449 21553 raft_consensus.cc:504] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.739917 21553 leader_election.cc:288] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:12.740478 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fd60c78334d90b26c3c9ce4945247" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:12.740736 20429 raft_consensus.cc:2454] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:12.741132 20497 leader_election.cc:302] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:12.741448 21553 raft_consensus.cc:2781] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:12.741541 21553 raft_consensus.cc:482] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:12.741652 21553 raft_consensus.cc:3037] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.743603 21553 raft_consensus.cc:504] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.744119 21553 leader_election.cc:288] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:12.744647 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fd60c78334d90b26c3c9ce4945247" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:12.744863 20429 raft_consensus.cc:3037] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.746953 20429 raft_consensus.cc:2454] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:12.747412 20497 leader_election.cc:302] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:12.747701 21553 raft_consensus.cc:2781] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:12.747870 21553 raft_consensus.cc:686] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:12.748144 21553 consensus_queue.cc:237] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } }
I20240502 21:26:12.750846 20287 catalog_manager.cc:5484] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:12.785215 21553 raft_consensus.cc:482] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.785418 21553 raft_consensus.cc:504] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.786054 21553 leader_election.cc:288] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:12.786784 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244" is_pre_election: true
I20240502 21:26:12.787035 20698 raft_consensus.cc:2454] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 0.
I20240502 21:26:12.787511 20496 leader_election.cc:302] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:12.787868 21553 raft_consensus.cc:2781] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:12.787987 21553 raft_consensus.cc:482] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:12.788081 21553 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.790179 21553 raft_consensus.cc:504] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.790742 21553 leader_election.cc:288] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:12.791307 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:12.791512 20698 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.793494 20698 raft_consensus.cc:2454] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 1.
I20240502 21:26:12.793974 20496 leader_election.cc:302] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:12.794327 21553 raft_consensus.cc:2781] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:12.794502 21553 raft_consensus.cc:686] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:12.794775 21553 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } }
I20240502 21:26:12.797834 20287 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:13.173482 21620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:13.173985 21620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:13.182296 21620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:13.194743 21553 consensus_queue.cc:1035] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 21:26:13.225811 21553 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:13.267458 21616 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 181.022 ms
time per row: 0.0905111 ms
I20240502 21:26:13.530630 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.4:46571
--local_ip_for_outbound_sockets=127.16.122.4
--tserver_master_addrs=127.16.122.62:40689
--webserver_port=45005
--webserver_interface=127.16.122.4
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:26:13.878340 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:13.878670 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:13.878809 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:13.886719 21671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:13.886852 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:13.887018 21671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.4
I20240502 21:26:13.897107 21671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.4:46571
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.122.4
--webserver_port=45005
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.4
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:13.898559 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:13.900297 21671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:13.913874 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:13.914537 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:13.915585 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:13.916776 21671 server_base.cc:1034] running on GCE node
I20240502 21:26:13.917551 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:13.918795 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:13.920068 21671 hybrid_clock.cc:648] HybridClock initialized: now 1714685173920008 us; error 55 us; skew 500 ppm
I20240502 21:26:13.920466 21671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:13.922865 21671 webserver.cc:460] Webserver started at http://127.16.122.4:45005/ using document root <none> and password file <none>
I20240502 21:26:13.923694 21671 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:13.923846 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:13.929066 21671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 21:26:13.931859 21685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:13.933207 21671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:13.933377 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "16745bde14bc4a43b532e6bb828f0625"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:13.934177 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:13.948824 21671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:13.949611 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:13.949898 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:13.950829 21671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:13.952351 21671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:13.952421 21671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:13.952548 21671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:13.952596 21671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:13.984148 21671 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.4:46571
I20240502 21:26:13.984230 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.4:46571 every 8 connection(s)
I20240502 21:26:13.985774 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:13.987496 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 21671
I20240502 21:26:13.988262 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.5:35311
--local_ip_for_outbound_sockets=127.16.122.5
--tserver_master_addrs=127.16.122.62:40689
--webserver_port=46015
--webserver_interface=127.16.122.5
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:14.011143 21798 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:14.011497 21798 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:14.012421 21798 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:14.014209 20280 ts_manager.cc:194] Re-registered known tserver with Master: 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571)
I20240502 21:26:14.015126 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.4:37061
W20240502 21:26:14.348031 21802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:14.348353 21802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:14.348502 21802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:14.356432 21802 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:14.356571 21802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:14.356748 21802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.5
I20240502 21:26:14.366330 21802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.5:35311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.122.5
--webserver_port=46015
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.5
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:14.367758 21802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:14.369498 21802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:14.382428 21807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:14.383543 21808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:14.384402 21810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:14.385572 21802 server_base.cc:1034] running on GCE node
I20240502 21:26:14.386219 21802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:14.387320 21802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:14.388538 21802 hybrid_clock.cc:648] HybridClock initialized: now 1714685174388480 us; error 64 us; skew 500 ppm
I20240502 21:26:14.388936 21802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:14.392103 21802 webserver.cc:460] Webserver started at http://127.16.122.5:46015/ using document root <none> and password file <none>
I20240502 21:26:14.393082 21802 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:14.393234 21802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:14.400367 21802 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:26:14.404129 21816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:14.405681 21802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:14.405885 21802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ee8dcd502996434ea55a16a62a5324fb"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-p9xn"
I20240502 21:26:14.407027 21802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:14.425455 21802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:14.426287 21802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:14.426594 21802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:14.427541 21802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:14.429047 21802 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:14.429119 21802 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:14.429260 21802 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:14.429311 21802 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:14.462235 21802 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.5:35311
I20240502 21:26:14.462278 21928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.5:35311 every 8 connection(s)
I20240502 21:26:14.463829 21802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:14.471648 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 21802
I20240502 21:26:14.472278 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.6:38855
--local_ip_for_outbound_sockets=127.16.122.6
--tserver_master_addrs=127.16.122.62:40689
--webserver_port=35735
--webserver_interface=127.16.122.6
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:14.477617 21929 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:14.477941 21929 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:14.478852 21929 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:14.480433 20280 ts_manager.cc:194] Re-registered known tserver with Master: ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311)
I20240502 21:26:14.481256 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.5:34297
W20240502 21:26:14.827244 21932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:14.827553 21932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:14.827679 21932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:14.836325 21932 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:14.836566 21932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:14.836803 21932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.6
I20240502 21:26:14.849231 21932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.6:38855
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.122.6
--webserver_port=35735
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.6
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:14.850677 21932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:14.852458 21932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:14.870873 21937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:14.873224 21938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:14.874626 21940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:14.875263 21932 server_base.cc:1034] running on GCE node
I20240502 21:26:14.875908 21932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:14.876958 21932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:14.878180 21932 hybrid_clock.cc:648] HybridClock initialized: now 1714685174878125 us; error 49 us; skew 500 ppm
I20240502 21:26:14.878554 21932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:14.881141 21932 webserver.cc:460] Webserver started at http://127.16.122.6:35735/ using document root <none> and password file <none>
I20240502 21:26:14.881896 21932 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:14.882018 21932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:14.887492 21932 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:14.890728 21946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:14.892707 21932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:26:14.892907 21932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c0aa2a3f325429eb83423bff3a80761"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:14.894109 21932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:14.915313 21932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:14.916172 21932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:14.916476 21932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:14.917515 21932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:14.919291 21932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:14.919366 21932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:14.919492 21932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:14.919541 21932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:14.975591 21932 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.6:38855
I20240502 21:26:14.975721 22058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.6:38855 every 8 connection(s)
I20240502 21:26:14.977494 21932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:14.981920 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 21932
I20240502 21:26:14.982723 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.7:34365
--local_ip_for_outbound_sockets=127.16.122.7
--tserver_master_addrs=127.16.122.62:40689
--webserver_port=37291
--webserver_interface=127.16.122.7
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:14.996766 22059 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:14.997129 22059 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:14.998080 22059 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:14.999899 20280 ts_manager.cc:194] Re-registered known tserver with Master: 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855)
I20240502 21:26:15.001343 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.6:59739
I20240502 21:26:15.017700 21798 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
W20240502 21:26:15.393770 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:15.394162 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:15.394374 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:15.404078 22064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:15.404227 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:15.404353 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.7
I20240502 21:26:15.413764 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.7:34365
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.122.7
--webserver_port=37291
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.7
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:15.415199 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:15.417083 22064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:15.430723 22072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:15.431257 22070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:15.432484 22069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:15.433650 22064 server_base.cc:1034] running on GCE node
I20240502 21:26:15.434306 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:15.435402 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:15.436717 22064 hybrid_clock.cc:648] HybridClock initialized: now 1714685175436662 us; error 48 us; skew 500 ppm
I20240502 21:26:15.437095 22064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:15.439755 22064 webserver.cc:460] Webserver started at http://127.16.122.7:37291/ using document root <none> and password file <none>
I20240502 21:26:15.440572 22064 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:15.440707 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:15.446736 22064 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:15.450016 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:15.451567 22064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:15.451764 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-p9xn"
I20240502 21:26:15.452720 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:15.484524 21929 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:15.489398 22064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:15.490397 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:15.490725 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:15.491762 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:15.493875 22085 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 21:26:15.499361 22064 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 21:26:15.499462 22064 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240502 21:26:15.499591 22064 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 21:26:15.502547 22064 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 21:26:15.502638 22064 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:15.503024 22085 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap starting.
I20240502 21:26:15.524174 22085 log.cc:824] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:15.535717 22085 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:15.536718 22085 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap complete.
I20240502 21:26:15.538094 22085 ts_tablet_manager.cc:1367] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.004s
I20240502 21:26:15.548606 22085 raft_consensus.cc:348] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:15.549861 22085 raft_consensus.cc:729] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.550679 22085 consensus_queue.cc:260] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:15.552479 22085 ts_tablet_manager.cc:1398] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20240502 21:26:15.558982 22064 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.7:34365
I20240502 21:26:15.559034 22193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.7:34365 every 8 connection(s)
I20240502 21:26:15.560446 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240502 21:26:15.561684 22195 tablet.cc:2369] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:15.567684 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 22064
I20240502 21:26:15.567981 16872 test_util.cc:268] Using random seed: 1087018569
I20240502 21:26:15.573741 22194 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:15.574142 22194 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:15.575094 22194 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:15.578099 20280 ts_manager.cc:194] Re-registered known tserver with Master: b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
I20240502 21:26:15.579733 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.7:35217
I20240502 21:26:15.585620 16872 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 21:26:15.586102 16872 test_util.cc:268] Using random seed: 1087036687
I20240502 21:26:15.602545 16872 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 21:26:15.602963 16872 test_util.cc:268] Using random seed: 1087053519
I20240502 21:26:15.626397 16872 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 21:26:15.626869 16872 test_util.cc:268] Using random seed: 1087077432
I20240502 21:26:15.682415 16872 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 21:26:15.688089 16872 test_util.cc:268] Using random seed: 1087138639
I20240502 21:26:15.853206 16872 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 21:26:16.005205 22059 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:16.583312 22194 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:16.995831 22319 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:16.996120 22319 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:16.997887 22319 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:17.054436 20426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e14655d22984ef3aab11be93eaaa07d" candidate_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7" is_pre_election: true
I20240502 21:26:17.058238 20426 raft_consensus.cc:2454] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4edc0e5b5c04ac5a7fe3f99d86d314b in term 1.
I20240502 21:26:17.061961 22082 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, b4edc0e5b5c04ac5a7fe3f99d86d314b; no voters:
I20240502 21:26:17.062785 22319 raft_consensus.cc:2781] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:26:17.062956 22319 raft_consensus.cc:482] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:17.063055 22319 raft_consensus.cc:3037] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:17.072786 22319 raft_consensus.cc:504] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:17.073437 22319 leader_election.cc:288] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 2 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:17.078388 20428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e14655d22984ef3aab11be93eaaa07d" candidate_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:17.078749 20428 raft_consensus.cc:3037] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:17.082407 20428 raft_consensus.cc:2454] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4edc0e5b5c04ac5a7fe3f99d86d314b in term 2.
I20240502 21:26:17.083143 22082 leader_election.cc:302] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, b4edc0e5b5c04ac5a7fe3f99d86d314b; no voters:
I20240502 21:26:17.083535 22319 raft_consensus.cc:2781] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:17.084419 22319 raft_consensus.cc:686] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Becoming Leader. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Running, Role: LEADER
I20240502 21:26:17.084975 22319 consensus_queue.cc:237] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } }
I20240502 21:26:17.138722 20281 catalog_manager.cc:5484] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:17.363250 20678 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:17.391813 21733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:17.411036 20408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:17.411947 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:17.436112 20544 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:17.503862 22127 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:17.525982 20426 raft_consensus.cc:1261] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:26:17.526875 22319 consensus_queue.cc:1035] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:17.604128 21864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:18.358431 20678 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:18.417358 21733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:18.475805 20544 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:18.591342 22127 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:18.592473 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:18.630234 20408 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:18.649447 21864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:18.784679 20698 consensus_queue.cc:237] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:18.796521 20425 raft_consensus.cc:1261] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 21:26:18.798990 22280 consensus_queue.cc:1035] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 21:26:18.805691 22280 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:18.806491 20429 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:18.819590 20278 catalog_manager.cc:5484] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index -1 to 65, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
W20240502 21:26:18.843119 20635 consensus_peers.cc:485] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855): Couldn't send request to peer 6c0aa2a3f325429eb83423bff3a80761. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 028bf9972bf44bbbad3b862c70b174ca. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:18.947495 20698 consensus_queue.cc:237] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:18.956352 20562 raft_consensus.cc:1261] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240502 21:26:18.957933 22223 consensus_queue.cc:1035] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240502 21:26:18.963497 22223 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:18.971158 20280 catalog_manager.cc:5484] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index -1 to 41, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:18.978446 20562 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
W20240502 21:26:19.021893 20632 consensus_peers.cc:485] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571): Couldn't send request to peer 16745bde14bc4a43b532e6bb828f0625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44116b77039245b4883892cf19a8a248. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.033092 20699 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:19.037585 20426 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 21:26:19.038283 22063 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
W20240502 21:26:19.045378 20635 consensus_peers.cc:485] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311): Couldn't send request to peer ee8dcd502996434ea55a16a62a5324fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2998322d830948ac858235f4fb7defca. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.046377 22063 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:19.051730 20429 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:19.061221 20280 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index -1 to 64, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:19.152565 20697 consensus_queue.cc:237] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:19.169237 20427 raft_consensus.cc:1261] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240502 21:26:19.170732 22280 consensus_queue.cc:1035] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
W20240502 21:26:19.179358 20631 consensus_peers.cc:485] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Couldn't send request to peer b4edc0e5b5c04ac5a7fe3f99d86d314b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef774f42c72349d0aa9d490bb5dd8011. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.182935 22318 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
I20240502 21:26:19.192765 20280 catalog_manager.cc:5484] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index -1 to 59, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:19.202687 20427 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
I20240502 21:26:19.221719 22451 ts_tablet_manager.cc:902] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:19.316444 22451 tablet_copy_client.cc:308] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:19.323844 20562 consensus_queue.cc:237] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:19.346308 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet 028bf9972bf44bbbad3b862c70b174ca from peer 6c0aa2a3f325429eb83423bff3a80761 ({username='slave'} at 127.16.122.6:48607)
I20240502 21:26:19.346719 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet 028bf9972bf44bbbad3b862c70b174ca from peer 6c0aa2a3f325429eb83423bff3a80761 at {username='slave'} at 127.16.122.6:48607: session id = 6c0aa2a3f325429eb83423bff3a80761-028bf9972bf44bbbad3b862c70b174ca
I20240502 21:26:19.349288 20426 raft_consensus.cc:1261] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 21:26:19.350786 22285 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 21:26:19.351384 20719 tablet_copy_source_session.cc:205] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:19.356380 22282 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:19.358069 22451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028bf9972bf44bbbad3b862c70b174ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.364284 20280 catalog_manager.cc:5484] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index -1 to 75, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:19.382799 22461 ts_tablet_manager.cc:902] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:19.387217 22451 tablet_copy_client.cc:783] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:19.387698 22451 tablet_copy_client.cc:647] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:19.392310 22451 tablet_copy_client.cc:523] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:19.394196 20429 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:19.404335 22451 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap starting.
W20240502 21:26:19.415582 20497 consensus_peers.cc:485] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf -> Peer 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855): Couldn't send request to peer 6c0aa2a3f325429eb83423bff3a80761. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0892742cff44146ae2fb345d7e2bd3a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.425369 22461 tablet_copy_client.cc:308] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:19.477617 22451 log.cc:824] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:19.488574 20697 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:19.491339 22473 ts_tablet_manager.cc:902] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:19.493525 20426 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 21:26:19.496717 22318 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
W20240502 21:26:19.509850 20632 consensus_peers.cc:485] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571): Couldn't send request to peer 16745bde14bc4a43b532e6bb828f0625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12887227115743afacb541c753db51c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.518172 22223 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:19.520820 22473 tablet_copy_client.cc:308] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:19.530473 20280 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index -1 to 74, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:19.531517 20429 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:19.615689 22482 ts_tablet_manager.cc:902] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:19.638792 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet 44116b77039245b4883892cf19a8a248 from peer 16745bde14bc4a43b532e6bb828f0625 ({username='slave'} at 127.16.122.4:44007)
I20240502 21:26:19.638983 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet 44116b77039245b4883892cf19a8a248 from peer 16745bde14bc4a43b532e6bb828f0625 at {username='slave'} at 127.16.122.4:44007: session id = 16745bde14bc4a43b532e6bb828f0625-44116b77039245b4883892cf19a8a248
I20240502 21:26:19.640815 22482 tablet_copy_client.cc:308] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:19.641144 20719 tablet_copy_source_session.cc:205] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:19.647835 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44116b77039245b4883892cf19a8a248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.649722 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet 2998322d830948ac858235f4fb7defca from peer ee8dcd502996434ea55a16a62a5324fb ({username='slave'} at 127.16.122.5:59733)
I20240502 21:26:19.649897 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet 2998322d830948ac858235f4fb7defca from peer ee8dcd502996434ea55a16a62a5324fb at {username='slave'} at 127.16.122.5:59733: session id = ee8dcd502996434ea55a16a62a5324fb-2998322d830948ac858235f4fb7defca
I20240502 21:26:19.677217 20719 tablet_copy_source_session.cc:205] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:19.683568 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998322d830948ac858235f4fb7defca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.695379 22461 tablet_copy_client.cc:783] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:19.697474 22461 tablet_copy_client.cc:647] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:19.705027 22461 tablet_copy_client.cc:523] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:19.707854 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet ef774f42c72349d0aa9d490bb5dd8011 from peer b4edc0e5b5c04ac5a7fe3f99d86d314b ({username='slave'} at 127.16.122.7:54293)
I20240502 21:26:19.712953 22461 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap starting.
I20240502 21:26:19.714303 20563 consensus_queue.cc:237] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:19.720957 22473 tablet_copy_client.cc:783] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:19.721297 22473 tablet_copy_client.cc:647] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:19.708076 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet ef774f42c72349d0aa9d490bb5dd8011 from peer b4edc0e5b5c04ac5a7fe3f99d86d314b at {username='slave'} at 127.16.122.7:54293: session id = b4edc0e5b5c04ac5a7fe3f99d86d314b-ef774f42c72349d0aa9d490bb5dd8011
I20240502 21:26:19.727702 22473 tablet_copy_client.cc:523] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:19.733078 22473 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Bootstrap starting.
W20240502 21:26:19.735675 20497 consensus_peers.cc:485] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf -> Peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311): Couldn't send request to peer ee8dcd502996434ea55a16a62a5324fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fabe357cd7d42259e10511baa31b0bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.745904 22461 log.cc:824] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:19.746887 20719 tablet_copy_source_session.cc:205] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:19.750694 20696 raft_consensus.cc:1261] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240502 21:26:19.758266 22482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef774f42c72349d0aa9d490bb5dd8011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.761020 22282 consensus_queue.cc:1035] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240502 21:26:19.767938 22482 tablet_copy_client.cc:783] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:19.768321 22482 tablet_copy_client.cc:647] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:19.773238 22451 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=706 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:19.773968 22451 tablet_bootstrap.cc:492] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap complete.
I20240502 21:26:19.774590 22451 ts_tablet_manager.cc:1367] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Time spent bootstrapping tablet: real 0.374s user 0.157s sys 0.011s
I20240502 21:26:19.782940 22451 raft_consensus.cc:348] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:19.783516 22451 raft_consensus.cc:729] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c0aa2a3f325429eb83423bff3a80761, State: Initialized, Role: LEARNER
I20240502 21:26:19.784143 22451 consensus_queue.cc:260] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:19.787323 22423 raft_consensus.cc:2932] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:19.798137 20280 catalog_manager.cc:5484] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index -1 to 76, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:19.812374 20696 raft_consensus.cc:2932] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:19.819154 22482 tablet_copy_client.cc:523] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:19.822599 22473 log.cc:824] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:19.824258 22482 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap starting.
I20240502 21:26:19.827857 22494 ts_tablet_manager.cc:902] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:19.881549 22451 ts_tablet_manager.cc:1398] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761: Time spent starting tablet: real 0.107s user 0.019s sys 0.018s
I20240502 21:26:19.884712 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session 6c0aa2a3f325429eb83423bff3a80761-028bf9972bf44bbbad3b862c70b174ca received from {username='slave'} at 127.16.122.6:48607
I20240502 21:26:19.884896 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session 6c0aa2a3f325429eb83423bff3a80761-028bf9972bf44bbbad3b862c70b174ca on tablet 028bf9972bf44bbbad3b862c70b174ca with peer 6c0aa2a3f325429eb83423bff3a80761
I20240502 21:26:19.958539 20696 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:19.960860 20564 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240502 21:26:19.961973 22318 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
W20240502 21:26:19.962924 20631 consensus_peers.cc:485] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Couldn't send request to peer b4edc0e5b5c04ac5a7fe3f99d86d314b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d15373bff59348b9a65b42053e8df389. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:19.970072 20564 raft_consensus.cc:1203] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.59] Dedup: 1.58->[1.59-1.59]
I20240502 21:26:19.984437 22223 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
I20240502 21:26:19.994030 20280 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index -1 to 59, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:19.995254 20564 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
I20240502 21:26:20.052578 22461 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=50 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:20.052870 22494 tablet_copy_client.cc:308] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:20.060715 22461 tablet_bootstrap.cc:492] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap complete.
I20240502 21:26:20.062157 22461 ts_tablet_manager.cc:1367] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Time spent bootstrapping tablet: real 0.350s user 0.153s sys 0.045s
I20240502 21:26:20.082165 22461 raft_consensus.cc:348] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:20.084172 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet d0892742cff44146ae2fb345d7e2bd3a from peer 6c0aa2a3f325429eb83423bff3a80761 ({username='slave'} at 127.16.122.6:56377)
I20240502 21:26:20.084515 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet d0892742cff44146ae2fb345d7e2bd3a from peer 6c0aa2a3f325429eb83423bff3a80761 at {username='slave'} at 127.16.122.6:56377: session id = 6c0aa2a3f325429eb83423bff3a80761-d0892742cff44146ae2fb345d7e2bd3a
I20240502 21:26:20.094004 22461 raft_consensus.cc:729] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16745bde14bc4a43b532e6bb828f0625, State: Initialized, Role: LEARNER
I20240502 21:26:20.089334 20585 tablet_copy_source_session.cc:205] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:20.095096 22461 consensus_queue.cc:260] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:20.096946 22494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0892742cff44146ae2fb345d7e2bd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.102583 22461 ts_tablet_manager.cc:1398] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625: Time spent starting tablet: real 0.039s user 0.017s sys 0.005s
I20240502 21:26:20.104665 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session 16745bde14bc4a43b532e6bb828f0625-44116b77039245b4883892cf19a8a248 received from {username='slave'} at 127.16.122.4:44007
I20240502 21:26:20.104820 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session 16745bde14bc4a43b532e6bb828f0625-44116b77039245b4883892cf19a8a248 on tablet 44116b77039245b4883892cf19a8a248 with peer 16745bde14bc4a43b532e6bb828f0625
I20240502 21:26:20.125604 22461 ts_tablet_manager.cc:902] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:20.169448 22494 tablet_copy_client.cc:783] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:20.169803 22494 tablet_copy_client.cc:647] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:20.190871 22494 tablet_copy_client.cc:523] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:20.193372 20563 consensus_queue.cc:237] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:20.195248 22494 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap starting.
I20240502 21:26:20.197707 20427 raft_consensus.cc:1261] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240502 21:26:20.198630 22423 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240502 21:26:20.203424 22285 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:20.207418 20425 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:20.217113 22461 tablet_copy_client.cc:308] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:20.217999 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet 12887227115743afacb541c753db51c9 from peer 16745bde14bc4a43b532e6bb828f0625 ({username='slave'} at 127.16.122.4:44007)
I20240502 21:26:20.218189 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet 12887227115743afacb541c753db51c9 from peer 16745bde14bc4a43b532e6bb828f0625 at {username='slave'} at 127.16.122.4:44007: session id = 16745bde14bc4a43b532e6bb828f0625-12887227115743afacb541c753db51c9
I20240502 21:26:20.220129 20280 catalog_manager.cc:5484] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index -1 to 61, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:20.221235 20719 tablet_copy_source_session.cc:205] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:20.222751 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12887227115743afacb541c753db51c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.243479 22461 tablet_copy_client.cc:783] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:20.243815 22461 tablet_copy_client.cc:647] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:20.254650 22461 tablet_copy_client.cc:523] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:20.289645 22461 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap starting.
I20240502 21:26:20.314129 22014 raft_consensus.cc:1203] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.64->[1.65-1.86] Dedup: 1.75->[1.76-1.86]
I20240502 21:26:20.329473 22148 consensus_queue.cc:237] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
W20240502 21:26:20.359216 20495 consensus_peers.cc:485] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf -> Peer 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571): Couldn't send request to peer 16745bde14bc4a43b532e6bb828f0625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 380fd60c78334d90b26c3c9ce4945247. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:20.383975 20426 raft_consensus.cc:1261] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 21:26:20.385147 22502 consensus_queue.cc:1035] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:20.389284 22529 ts_tablet_manager.cc:902] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
W20240502 21:26:20.404824 22082 consensus_peers.cc:485] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b -> Peer 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855): Couldn't send request to peer 6c0aa2a3f325429eb83423bff3a80761. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e14655d22984ef3aab11be93eaaa07d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:20.416190 22539 ts_tablet_manager.cc:902] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:20.438024 22223 raft_consensus.cc:1050] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 to VOTER
I20240502 21:26:20.439949 22223 consensus_queue.cc:237] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:20.440564 22529 tablet_copy_client.cc:308] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:20.442948 20429 raft_consensus.cc:1261] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 21:26:20.442981 22014 raft_consensus.cc:1261] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 21:26:20.443753 22318 consensus_queue.cc:1035] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240502 21:26:20.444802 22318 consensus_queue.cc:1035] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240502 21:26:20.447538 22280 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 65 to 88, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.451122 21753 raft_consensus.cc:1203] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.60] Dedup: 1.51->[1.52-1.60]
I20240502 21:26:20.454067 22014 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 65 to 88, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.456707 20278 catalog_manager.cc:5484] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 65 to 88, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:20.458051 20429 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 65 to 88, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.462865 20696 consensus_queue.cc:237] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:20.476475 22482 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:20.481351 22013 raft_consensus.cc:1261] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240502 21:26:20.485740 22223 consensus_queue.cc:1035] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240502 21:26:20.493455 22223 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index 88 to 89, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.494139 22482 tablet_bootstrap.cc:492] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap complete.
I20240502 21:26:20.494731 22482 ts_tablet_manager.cc:1367] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent bootstrapping tablet: real 0.671s user 0.136s sys 0.015s
I20240502 21:26:20.496409 22482 raft_consensus.cc:348] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:20.500118 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet 3fabe357cd7d42259e10511baa31b0bc from peer ee8dcd502996434ea55a16a62a5324fb ({username='slave'} at 127.16.122.5:48075)
I20240502 21:26:20.500396 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet 3fabe357cd7d42259e10511baa31b0bc from peer ee8dcd502996434ea55a16a62a5324fb at {username='slave'} at 127.16.122.5:48075: session id = ee8dcd502996434ea55a16a62a5324fb-3fabe357cd7d42259e10511baa31b0bc
I20240502 21:26:20.505968 20585 tablet_copy_source_session.cc:205] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:20.507980 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fabe357cd7d42259e10511baa31b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.508692 22014 raft_consensus.cc:2932] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 88 to 89, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.516113 20278 catalog_manager.cc:5484] T 028bf9972bf44bbbad3b862c70b174ca P 6c0aa2a3f325429eb83423bff3a80761 reported cstate change: config changed from index 88 to 89, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
W20240502 21:26:20.519083 22473 log.cc:924] Time spent T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20240502 21:26:20.525238 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet 028bf9972bf44bbbad3b862c70b174ca with delete_type TABLET_DATA_TOMBSTONED (TS 35812e2216cb4758aee1e94877a823b7 not found in new config with opid_index 89) from {username='slave'} at 127.0.0.1:34202
I20240502 21:26:20.527155 22482 raft_consensus.cc:729] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Initialized, Role: LEARNER
I20240502 21:26:20.529057 22482 consensus_queue.cc:260] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:20.531173 22482 ts_tablet_manager.cc:1398] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20240502 21:26:20.532893 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session b4edc0e5b5c04ac5a7fe3f99d86d314b-ef774f42c72349d0aa9d490bb5dd8011 received from {username='slave'} at 127.16.122.7:54293
I20240502 21:26:20.533088 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session b4edc0e5b5c04ac5a7fe3f99d86d314b-ef774f42c72349d0aa9d490bb5dd8011 on tablet ef774f42c72349d0aa9d490bb5dd8011 with peer b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:20.524873 22473 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:20.528594 22529 tablet_copy_client.cc:783] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:20.540766 22529 tablet_copy_client.cc:647] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:20.541118 22473 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Bootstrap complete.
I20240502 21:26:20.541705 22473 ts_tablet_manager.cc:1367] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Time spent bootstrapping tablet: real 0.809s user 0.183s sys 0.010s
I20240502 21:26:20.543141 22529 tablet_copy_client.cc:523] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:20.546319 20273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 028bf9972bf44bbbad3b862c70b174ca with cas_config_opid_index 88: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:20.556222 22529 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: Bootstrap starting.
I20240502 21:26:20.585279 22513 ts_tablet_manager.cc:902] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:20.597724 22553 tablet_replica.cc:330] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: stopping tablet replica
I20240502 21:26:20.598419 22553 raft_consensus.cc:2229] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:20.599552 22553 raft_consensus.cc:2258] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:20.600672 22539 tablet_copy_client.cc:308] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:20.602325 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet d15373bff59348b9a65b42053e8df389 from peer b4edc0e5b5c04ac5a7fe3f99d86d314b ({username='slave'} at 127.16.122.7:54293)
I20240502 21:26:20.602576 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet d15373bff59348b9a65b42053e8df389 from peer b4edc0e5b5c04ac5a7fe3f99d86d314b at {username='slave'} at 127.16.122.7:54293: session id = b4edc0e5b5c04ac5a7fe3f99d86d314b-d15373bff59348b9a65b42053e8df389
I20240502 21:26:20.605831 20719 tablet_copy_source_session.cc:205] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:20.609782 22539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15373bff59348b9a65b42053e8df389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.621757 22539 tablet_copy_client.cc:783] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:20.622148 22539 tablet_copy_client.cc:647] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:20.633311 22553 ts_tablet_manager.cc:1875] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:20.639005 22539 tablet_copy_client.cc:523] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:20.645793 22539 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap starting.
I20240502 21:26:20.654651 22558 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:20.656138 22494 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap replayed 1/1 log segments. Stats: ops{read=82 overwritten=0 applied=82 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:20.656715 22494 tablet_bootstrap.cc:492] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap complete.
I20240502 21:26:20.657095 22494 ts_tablet_manager.cc:1367] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: Time spent bootstrapping tablet: real 0.462s user 0.164s sys 0.030s
I20240502 21:26:20.658319 22494 raft_consensus.cc:348] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:20.662595 22494 raft_consensus.cc:729] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c0aa2a3f325429eb83423bff3a80761, State: Initialized, Role: LEARNER
I20240502 21:26:20.662946 22494 consensus_queue.cc:260] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.82, Last appended by leader: 82, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:20.663873 22494 ts_tablet_manager.cc:1398] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:26:20.665104 20585 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session 6c0aa2a3f325429eb83423bff3a80761-d0892742cff44146ae2fb345d7e2bd3a received from {username='slave'} at 127.16.122.6:56377
I20240502 21:26:20.665329 20585 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session 6c0aa2a3f325429eb83423bff3a80761-d0892742cff44146ae2fb345d7e2bd3a on tablet d0892742cff44146ae2fb345d7e2bd3a with peer 6c0aa2a3f325429eb83423bff3a80761
I20240502 21:26:20.685237 20278 catalog_manager.cc:5484] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index -1 to 3, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:20.686944 20426 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:20.700646 22473 raft_consensus.cc:348] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:20.747987 22513 tablet_copy_client.cc:308] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:20.760084 22473 raft_consensus.cc:729] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Initialized, Role: LEARNER
I20240502 21:26:20.766410 22473 consensus_queue.cc:260] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:20.774495 22063 raft_consensus.cc:1050] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER 16745bde14bc4a43b532e6bb828f0625 to VOTER
I20240502 21:26:20.775350 22063 consensus_queue.cc:237] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:20.778935 21753 raft_consensus.cc:1261] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240502 21:26:20.782155 22318 consensus_queue.cc:1035] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240502 21:26:20.790381 22280 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index 41 to 61, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:20.792042 20563 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
W20240502 21:26:20.795392 20497 consensus_peers.cc:485] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf -> Peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311): Couldn't send request to peer ee8dcd502996434ea55a16a62a5324fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4aeebd7828643ccab4aa62d4fd4d12a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:20.797843 22014 raft_consensus.cc:1203] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.74->[1.75-1.92] Dedup: 1.82->[1.83-1.92]
I20240502 21:26:20.803820 20280 catalog_manager.cc:5484] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 41 to 61, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:20.804352 21753 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index 41 to 61, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:20.812881 22473 ts_tablet_manager.cc:1398] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Time spent starting tablet: real 0.271s user 0.012s sys 0.000s
I20240502 21:26:20.815171 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session ee8dcd502996434ea55a16a62a5324fb-2998322d830948ac858235f4fb7defca received from {username='slave'} at 127.16.122.5:59733
I20240502 21:26:20.815331 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session ee8dcd502996434ea55a16a62a5324fb-2998322d830948ac858235f4fb7defca on tablet 2998322d830948ac858235f4fb7defca with peer ee8dcd502996434ea55a16a62a5324fb
I20240502 21:26:20.819860 20696 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 21:26:20.822183 22553 ts_tablet_manager.cc:1888] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.88
I20240502 21:26:20.822428 22553 log.cc:1196] T 028bf9972bf44bbbad3b862c70b174ca P 35812e2216cb4758aee1e94877a823b7: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/028bf9972bf44bbbad3b862c70b174ca
I20240502 21:26:20.823731 20563 raft_consensus.cc:1261] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240502 21:26:20.824162 22282 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 21:26:20.827765 22063 consensus_queue.cc:1035] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240502 21:26:20.838304 20274 catalog_manager.cc:4830] TS 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): tablet 028bf9972bf44bbbad3b862c70b174ca (table rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]) successfully deleted
I20240502 21:26:20.844542 20696 consensus_queue.cc:237] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:20.845749 22285 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:20.852874 20278 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index -1 to 66, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:20.860968 21753 raft_consensus.cc:1261] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240502 21:26:20.861775 22318 consensus_queue.cc:1035] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240502 21:26:20.866582 22280 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index 61 to 63, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:20.868726 21753 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index 61 to 63, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:20.871249 20563 raft_consensus.cc:2932] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index 41 to 61, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:20.874936 22461 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=81 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:20.875504 22461 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap complete.
I20240502 21:26:20.875892 22461 ts_tablet_manager.cc:1367] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: Time spent bootstrapping tablet: real 0.597s user 0.137s sys 0.021s
I20240502 21:26:20.877976 22461 raft_consensus.cc:348] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:20.878037 20280 catalog_manager.cc:5484] T 44116b77039245b4883892cf19a8a248 P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: config changed from index 61 to 63, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:20.878325 22461 raft_consensus.cc:729] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16745bde14bc4a43b532e6bb828f0625, State: Initialized, Role: LEARNER
I20240502 21:26:20.878650 22461 consensus_queue.cc:260] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 81, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:20.879362 22461 ts_tablet_manager.cc:1398] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:20.880429 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session 16745bde14bc4a43b532e6bb828f0625-12887227115743afacb541c753db51c9 received from {username='slave'} at 127.16.122.4:44007
I20240502 21:26:20.880569 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session 16745bde14bc4a43b532e6bb828f0625-12887227115743afacb541c753db51c9 on tablet 12887227115743afacb541c753db51c9 with peer 16745bde14bc4a43b532e6bb828f0625
I20240502 21:26:20.882784 22281 raft_consensus.cc:1050] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 to VOTER
I20240502 21:26:20.883777 22281 consensus_queue.cc:237] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:20.887856 22014 raft_consensus.cc:1261] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:20.890128 22423 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 21:26:20.893793 20273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44116b77039245b4883892cf19a8a248 with cas_config_opid_index 61: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:20.901042 22494 ts_tablet_manager.cc:902] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: Initiating tablet copy from peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
I20240502 21:26:20.902901 22285 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 75 to 94, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.905180 22014 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 75 to 94, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.911522 20284 catalog_manager.cc:5484] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 reported cstate change: config changed from index 75 to 94, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.919637 22148 raft_consensus.cc:1203] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.87] Dedup: 1.70->[1.71-1.87]
I20240502 21:26:20.926256 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet 380fd60c78334d90b26c3c9ce4945247 from peer 16745bde14bc4a43b532e6bb828f0625 ({username='slave'} at 127.16.122.4:41433)
I20240502 21:26:20.926491 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet 380fd60c78334d90b26c3c9ce4945247 from peer 16745bde14bc4a43b532e6bb828f0625 at {username='slave'} at 127.16.122.4:41433: session id = 16745bde14bc4a43b532e6bb828f0625-380fd60c78334d90b26c3c9ce4945247
I20240502 21:26:20.929824 22494 tablet_copy_client.cc:308] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.7:34365
I20240502 21:26:20.929971 20585 tablet_copy_source_session.cc:205] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:20.932379 20696 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:20.945348 20426 raft_consensus.cc:1261] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:20.952716 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet 44116b77039245b4883892cf19a8a248 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 61 (current committed config index is 63)) from {username='slave'} at 127.0.0.1:34210
I20240502 21:26:20.956560 22513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380fd60c78334d90b26c3c9ce4945247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.963452 22513 tablet_copy_client.cc:783] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:20.963780 22513 tablet_copy_client.cc:647] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:20.963670 22282 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 21:26:20.969874 20425 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 75 to 94, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:20.974449 22585 tablet_replica.cc:330] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:20.975066 22585 raft_consensus.cc:2229] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:20.976027 22585 raft_consensus.cc:2258] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:20.978389 22585 ts_tablet_manager.cc:1875] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:20.990737 22513 tablet_copy_client.cc:523] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:20.998227 22168 tablet_copy_service.cc:140] P b4edc0e5b5c04ac5a7fe3f99d86d314b: Received BeginTabletCopySession request for tablet 0e14655d22984ef3aab11be93eaaa07d from peer 6c0aa2a3f325429eb83423bff3a80761 ({username='slave'} at 127.16.122.6:34427)
I20240502 21:26:20.998603 22168 tablet_copy_service.cc:161] P b4edc0e5b5c04ac5a7fe3f99d86d314b: Beginning new tablet copy session on tablet 0e14655d22984ef3aab11be93eaaa07d from peer 6c0aa2a3f325429eb83423bff3a80761 at {username='slave'} at 127.16.122.6:34427: session id = 6c0aa2a3f325429eb83423bff3a80761-0e14655d22984ef3aab11be93eaaa07d
I20240502 21:26:21.002239 22168 tablet_copy_source_session.cc:205] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:21.005039 22494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e14655d22984ef3aab11be93eaaa07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:21.012368 22585 ts_tablet_manager.cc:1888] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.62
I20240502 21:26:21.012671 22585 log.cc:1196] T 44116b77039245b4883892cf19a8a248 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/44116b77039245b4883892cf19a8a248
I20240502 21:26:21.013150 22494 tablet_copy_client.cc:783] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:21.013383 22494 tablet_copy_client.cc:647] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:21.016634 22494 tablet_copy_client.cc:523] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:21.018421 22513 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap starting.
I20240502 21:26:21.025126 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet 44116b77039245b4883892cf19a8a248 (table rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]) successfully deleted
I20240502 21:26:21.029377 22494 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap starting.
I20240502 21:26:21.044409 22494 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:21.044936 22494 tablet_bootstrap.cc:492] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap complete.
I20240502 21:26:21.045295 22494 ts_tablet_manager.cc:1367] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240502 21:26:21.046279 22494 raft_consensus.cc:348] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:21.046736 22494 raft_consensus.cc:729] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6c0aa2a3f325429eb83423bff3a80761, State: Initialized, Role: LEARNER
I20240502 21:26:21.047021 22494 consensus_queue.cc:260] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:21.057570 22494 ts_tablet_manager.cc:1398] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 21:26:21.062784 22168 tablet_copy_service.cc:342] P b4edc0e5b5c04ac5a7fe3f99d86d314b: Request end of tablet copy session 6c0aa2a3f325429eb83423bff3a80761-0e14655d22984ef3aab11be93eaaa07d received from {username='slave'} at 127.16.122.6:34427
I20240502 21:26:21.063004 22168 tablet_copy_service.cc:433] P b4edc0e5b5c04ac5a7fe3f99d86d314b: ending tablet copy session 6c0aa2a3f325429eb83423bff3a80761-0e14655d22984ef3aab11be93eaaa07d on tablet 0e14655d22984ef3aab11be93eaaa07d with peer 6c0aa2a3f325429eb83423bff3a80761
I20240502 21:26:21.095185 20563 consensus_queue.cc:237] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
W20240502 21:26:21.099392 20497 consensus_peers.cc:485] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf -> Peer 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855): Couldn't send request to peer 6c0aa2a3f325429eb83423bff3a80761. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bdf4a8993904b02b2dd0fe3ba0f9e4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:21.112155 22539 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap replayed 1/1 log segments. Stats: ops{read=62 overwritten=0 applied=62 ignored=0} inserts{seen=693 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:21.116865 20696 raft_consensus.cc:1261] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 21:26:21.118749 22539 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap complete.
I20240502 21:26:21.124636 22539 ts_tablet_manager.cc:1367] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent bootstrapping tablet: real 0.473s user 0.125s sys 0.009s
I20240502 21:26:21.127832 22539 raft_consensus.cc:348] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:21.128124 22539 raft_consensus.cc:729] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Initialized, Role: LEARNER
I20240502 21:26:21.129276 22539 consensus_queue.cc:260] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 62, Last appended: 1.62, Last appended by leader: 62, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:21.129663 22223 raft_consensus.cc:1050] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b to VOTER
I20240502 21:26:21.129869 22539 ts_tablet_manager.cc:1398] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:21.132951 22223 consensus_queue.cc:237] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:21.136363 22147 raft_consensus.cc:1261] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 21:26:21.142891 22318 consensus_queue.cc:1035] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240502 21:26:21.146643 20427 raft_consensus.cc:1261] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 21:26:21.146632 21884 raft_consensus.cc:1203] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.63->[1.64-1.82] Dedup: 1.75->[1.76-1.82]
I20240502 21:26:21.147933 22223 consensus_queue.cc:1035] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 21:26:21.144876 22148 raft_consensus.cc:1203] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.91->[1.92-1.93] Dedup: 1.92->[1.93-1.93]
I20240502 21:26:21.150458 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session b4edc0e5b5c04ac5a7fe3f99d86d314b-d15373bff59348b9a65b42053e8df389 received from {username='slave'} at 127.16.122.7:54293
I20240502 21:26:21.150616 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session b4edc0e5b5c04ac5a7fe3f99d86d314b-d15373bff59348b9a65b42053e8df389 on tablet d15373bff59348b9a65b42053e8df389 with peer b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:21.152029 22547 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 59 to 93, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.154014 22148 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 59 to 93, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.159150 20280 catalog_manager.cc:5484] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 59 to 93, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:21.166956 20427 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 59 to 93, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.169692 22529 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=81 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:21.171749 22529 tablet_bootstrap.cc:492] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: Bootstrap complete.
I20240502 21:26:21.172852 22529 ts_tablet_manager.cc:1367] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: Time spent bootstrapping tablet: real 0.617s user 0.135s sys 0.032s
I20240502 21:26:21.174729 22529 raft_consensus.cc:348] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:21.177071 22529 raft_consensus.cc:729] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Initialized, Role: LEARNER
I20240502 21:26:21.177350 22529 consensus_queue.cc:260] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 81, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:21.177503 20696 consensus_queue.cc:237] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:21.178051 21753 raft_consensus.cc:1203] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.99] Dedup: 1.81->[1.82-1.99]
I20240502 21:26:21.179399 22148 raft_consensus.cc:1261] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:21.182030 22529 ts_tablet_manager.cc:1398] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:26:21.183230 22318 consensus_queue.cc:1035] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 21:26:21.187587 20585 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session ee8dcd502996434ea55a16a62a5324fb-3fabe357cd7d42259e10511baa31b0bc received from {username='slave'} at 127.16.122.5:48075
I20240502 21:26:21.187777 20585 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session ee8dcd502996434ea55a16a62a5324fb-3fabe357cd7d42259e10511baa31b0bc on tablet 3fabe357cd7d42259e10511baa31b0bc with peer ee8dcd502996434ea55a16a62a5324fb
I20240502 21:26:21.189533 22547 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.189808 22148 raft_consensus.cc:2932] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.197121 20281 catalog_manager.cc:5484] T ef774f42c72349d0aa9d490bb5dd8011 P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 93 to 94, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.202090 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet ef774f42c72349d0aa9d490bb5dd8011 with delete_type TABLET_DATA_TOMBSTONED (TS 35812e2216cb4758aee1e94877a823b7 not found in new config with opid_index 94) from {username='slave'} at 127.0.0.1:34202
I20240502 21:26:21.202558 22553 tablet_replica.cc:330] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: stopping tablet replica
I20240502 21:26:21.202778 22553 raft_consensus.cc:2229] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:21.203176 22553 raft_consensus.cc:2258] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:21.204555 22553 ts_tablet_manager.cc:1875] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:21.205703 22529 ts_tablet_manager.cc:902] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:21.205850 22423 consensus_queue.cc:1035] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 21:26:21.217020 22553 ts_tablet_manager.cc:1888] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.93
I20240502 21:26:21.217350 22553 log.cc:1196] T ef774f42c72349d0aa9d490bb5dd8011 P 35812e2216cb4758aee1e94877a823b7: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef774f42c72349d0aa9d490bb5dd8011
I20240502 21:26:21.221562 22423 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:21.231308 20281 catalog_manager.cc:5484] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index -1 to 58, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:21.237165 20274 catalog_manager.cc:4830] TS 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): tablet ef774f42c72349d0aa9d490bb5dd8011 (table rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]) successfully deleted
I20240502 21:26:21.240072 20273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef774f42c72349d0aa9d490bb5dd8011 with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:21.244732 20698 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } } }
I20240502 21:26:21.267354 22063 raft_consensus.cc:1050] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER ee8dcd502996434ea55a16a62a5324fb to VOTER
I20240502 21:26:21.268219 22063 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:21.270917 20426 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 21:26:21.272112 22280 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 21:26:21.272962 21884 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 21:26:21.273669 22063 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 21:26:21.275456 22148 raft_consensus.cc:1203] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.69] Dedup: 1.62->[1.63-1.69]
I20240502 21:26:21.276893 22529 tablet_copy_client.cc:308] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:21.277954 22318 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 64 to 84, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.280408 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet d4aeebd7828643ccab4aa62d4fd4d12a from peer ee8dcd502996434ea55a16a62a5324fb ({username='slave'} at 127.16.122.5:48075)
I20240502 21:26:21.280611 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet d4aeebd7828643ccab4aa62d4fd4d12a from peer ee8dcd502996434ea55a16a62a5324fb at {username='slave'} at 127.16.122.5:48075: session id = ee8dcd502996434ea55a16a62a5324fb-d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:21.283493 20585 tablet_copy_source_session.cc:205] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:21.287595 20426 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 64 to 84, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.294287 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aeebd7828643ccab4aa62d4fd4d12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:21.295333 22014 raft_consensus.cc:1203] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 21:26:21.296818 20287 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index 64 to 84, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.317107 21884 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 64 to 84, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.318488 22529 tablet_copy_client.cc:783] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:21.318814 22529 tablet_copy_client.cc:647] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:21.328900 22529 tablet_copy_client.cc:523] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:21.329478 20564 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
W20240502 21:26:21.331997 20495 consensus_peers.cc:485] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf -> Peer 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571): Couldn't send request to peer 16745bde14bc4a43b532e6bb828f0625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ca7cd6b72d4419aade387eaf375454e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:21.333014 22547 raft_consensus.cc:1050] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER 16745bde14bc4a43b532e6bb828f0625 to VOTER
I20240502 21:26:21.333998 22529 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Bootstrap starting.
I20240502 21:26:21.342625 20696 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240502 21:26:21.333885 22547 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.100, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:21.353156 22281 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 21:26:21.355086 22318 raft_consensus.cc:1013] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: attempt to promote peer 16745bde14bc4a43b532e6bb828f0625: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:26:21.356616 21753 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240502 21:26:21.358546 22318 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 21:26:21.359387 22613 raft_consensus.cc:1050] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b to VOTER
I20240502 21:26:21.360026 22613 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:21.363071 22148 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 21:26:21.362762 22223 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.101: config changed from index 74 to 101, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.363701 22615 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 21:26:21.366305 22615 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 59 to 70, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.367106 21753 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 74 to 101, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.373809 20281 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: config changed from index 74 to 101, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.374163 20564 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 21:26:21.374997 22613 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 21:26:21.376294 22602 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:21.379472 22148 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 59 to 70, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.380335 22513 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:21.380774 22513 tablet_bootstrap.cc:492] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: Bootstrap complete.
I20240502 21:26:21.381067 22513 ts_tablet_manager.cc:1367] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: Time spent bootstrapping tablet: real 0.363s user 0.127s sys 0.028s
I20240502 21:26:21.381896 22513 raft_consensus.cc:348] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:21.382158 22513 raft_consensus.cc:729] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16745bde14bc4a43b532e6bb828f0625, State: Initialized, Role: LEARNER
I20240502 21:26:21.383061 22513 consensus_queue.cc:260] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:21.387144 20425 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240502 21:26:21.387143 20287 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 59 to 70, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.389293 22223 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 21:26:21.390777 20425 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 74 to 101, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.393069 20696 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } } }
I20240502 21:26:21.395462 20280 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index -1 to 76, NON_VOTER 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:21.398913 22513 ts_tablet_manager.cc:1398] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20240502 21:26:21.401743 20564 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 59 to 70, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:21.407395 20585 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session 16745bde14bc4a43b532e6bb828f0625-380fd60c78334d90b26c3c9ce4945247 received from {username='slave'} at 127.16.122.4:41433
I20240502 21:26:21.407598 20585 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session 16745bde14bc4a43b532e6bb828f0625-380fd60c78334d90b26c3c9ce4945247 on tablet 380fd60c78334d90b26c3c9ce4945247 with peer 16745bde14bc4a43b532e6bb828f0625
I20240502 21:26:21.450778 20697 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:21.453562 21753 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 21:26:21.454681 22547 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 21:26:21.460852 22615 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 101 to 104, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.472082 20273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12887227115743afacb541c753db51c9 with cas_config_opid_index 101: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:21.471364 20281 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 101 to 104, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:21.472586 21753 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 101 to 104, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.473416 21884 raft_consensus.cc:1203] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.75->[1.76-1.84] Dedup: 1.81->[1.82-1.84]
I20240502 21:26:21.488966 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet 12887227115743afacb541c753db51c9 with delete_type TABLET_DATA_TOMBSTONED (TS 35812e2216cb4758aee1e94877a823b7 not found in new config with opid_index 104) from {username='slave'} at 127.0.0.1:34202
I20240502 21:26:21.489388 22553 tablet_replica.cc:330] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: stopping tablet replica
I20240502 21:26:21.489601 22553 raft_consensus.cc:2229] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:21.489840 22553 raft_consensus.cc:2258] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:21.491202 22553 ts_tablet_manager.cc:1875] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:21.511561 22553 ts_tablet_manager.cc:1888] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.103
I20240502 21:26:21.529548 22553 log.cc:1196] T 12887227115743afacb541c753db51c9 P 35812e2216cb4758aee1e94877a823b7: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/12887227115743afacb541c753db51c9
I20240502 21:26:21.530992 20274 catalog_manager.cc:4830] TS 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): tablet 12887227115743afacb541c753db51c9 (table rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]) successfully deleted
I20240502 21:26:21.547611 22282 raft_consensus.cc:1050] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER ee8dcd502996434ea55a16a62a5324fb to VOTER
I20240502 21:26:21.548696 22282 consensus_queue.cc:237] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:21.552608 21884 raft_consensus.cc:1261] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 21:26:21.555687 22602 consensus_queue.cc:1035] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 21:26:21.569737 22281 raft_consensus.cc:2932] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index 76 to 86, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.572870 21883 raft_consensus.cc:2932] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 76 to 86, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.579736 20280 catalog_manager.cc:5484] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 76 to 86, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:21.587675 20564 consensus_queue.cc:237] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:21.590134 21883 raft_consensus.cc:1261] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 21:26:21.590934 22281 consensus_queue.cc:1035] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 21:26:21.594045 22285 raft_consensus.cc:2932] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 86 to 88, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.599562 20272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fabe357cd7d42259e10511baa31b0bc with cas_config_opid_index 86: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:21.601101 20280 catalog_manager.cc:5484] T 3fabe357cd7d42259e10511baa31b0bc P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 86 to 88, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:21.607524 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fabe357cd7d42259e10511baa31b0bc with delete_type TABLET_DATA_TOMBSTONED (TS 0ac8c8b961dc4500bc8b9f1055298244 not found in new config with opid_index 88) from {username='slave'} at 127.0.0.1:50304
I20240502 21:26:21.615638 21883 raft_consensus.cc:2932] T 3fabe357cd7d42259e10511baa31b0bc P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 86 to 88, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.619133 22627 tablet_replica.cc:330] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: stopping tablet replica
I20240502 21:26:21.619944 22627 raft_consensus.cc:2229] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:21.620836 22627 pending_rounds.cc:62] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Trying to abort 1 pending ops.
I20240502 21:26:21.621068 22627 pending_rounds.cc:68] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Aborting op as it isn't in flight: id { term: 1 index: 85 } timestamp: 7023350503481028608 op_type: WRITE_OP write_request { tablet_id: "3fabe357cd7d42259e10511baa31b0bc" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\010\363F}7%\005\274\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5f92cbd5bd087bf0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350503425855488 authz_token { token_data: "\010\243\214\320\261\006\"3\n\005slave\022*\n b813afcdc0bb448499de47644d610fac\020\001\030\001 \001(\001" signature: "<redacted>""\013\277\003mv\377\215<\023\017\274\322\206\205\270\271\255 !\013\256N\025}aN\341\225\366\024\n\351q7\335\246\020\321E\215\300\2162.\226\222T^\277(R}\0306\253\0371L\023\354\377\332\232\220" signing_key_seq_num: 0 } } request_id { client_id: "3900c5d1122e4f78a18eac99a9cebc91" seq_no: 245 first_incomplete_seq_no: 245 attempt_no: 0 }
I20240502 21:26:21.622457 22627 raft_consensus.cc:2258] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:21.625614 22627 ts_tablet_manager.cc:1875] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:21.636234 22627 ts_tablet_manager.cc:1888] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.85
I20240502 21:26:21.636457 22627 log.cc:1196] T 3fabe357cd7d42259e10511baa31b0bc P 0ac8c8b961dc4500bc8b9f1055298244: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/3fabe357cd7d42259e10511baa31b0bc
I20240502 21:26:21.637567 20273 catalog_manager.cc:4830] TS 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): tablet 3fabe357cd7d42259e10511baa31b0bc (table rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]) successfully deleted
I20240502 21:26:21.650142 22529 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=68 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:21.650784 22529 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Bootstrap complete.
I20240502 21:26:21.651093 22529 ts_tablet_manager.cc:1367] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Time spent bootstrapping tablet: real 0.317s user 0.137s sys 0.021s
I20240502 21:26:21.652993 22529 raft_consensus.cc:348] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:21.653236 22529 raft_consensus.cc:729] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Initialized, Role: LEARNER
I20240502 21:26:21.653445 21753 raft_consensus.cc:1203] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.60->[1.61-1.67] Dedup: 1.64->[1.65-1.67]
I20240502 21:26:21.653551 22529 consensus_queue.cc:260] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:21.655387 22529 ts_tablet_manager.cc:1398] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:26:21.656092 20585 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session ee8dcd502996434ea55a16a62a5324fb-d4aeebd7828643ccab4aa62d4fd4d12a received from {username='slave'} at 127.16.122.5:48075
I20240502 21:26:21.656237 20585 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session ee8dcd502996434ea55a16a62a5324fb-d4aeebd7828643ccab4aa62d4fd4d12a on tablet d4aeebd7828643ccab4aa62d4fd4d12a with peer ee8dcd502996434ea55a16a62a5324fb
I20240502 21:26:21.717938 22629 ts_tablet_manager.cc:902] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:21.729305 22282 raft_consensus.cc:1050] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER 16745bde14bc4a43b532e6bb828f0625 to VOTER
I20240502 21:26:21.730024 22282 consensus_queue.cc:237] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:21.734077 21753 raft_consensus.cc:1261] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 21:26:21.734997 22593 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 21:26:21.737780 22629 tablet_copy_client.cc:308] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:21.739039 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f from peer 6c0aa2a3f325429eb83423bff3a80761 ({username='slave'} at 127.16.122.6:56377)
I20240502 21:26:21.739218 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f from peer 6c0aa2a3f325429eb83423bff3a80761 at {username='slave'} at 127.16.122.6:56377: session id = 6c0aa2a3f325429eb83423bff3a80761-8bdf4a8993904b02b2dd0fe3ba0f9e4f
I20240502 21:26:21.742053 20585 tablet_copy_source_session.cc:205] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:21.741981 22593 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 61 to 70, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.742457 20426 raft_consensus.cc:1261] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 21:26:21.742456 21753 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 61 to 70, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.743697 22285 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 21:26:21.745807 20426 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 61 to 70, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.749394 20281 catalog_manager.cc:5484] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: config changed from index 61 to 70, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:21.752789 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:21.760118 22629 tablet_copy_client.cc:783] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:21.760469 22629 tablet_copy_client.cc:647] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:21.780083 21883 raft_consensus.cc:1203] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69] Dedup: 1.68->[1.69-1.69]
I20240502 21:26:21.780519 22629 tablet_copy_client.cc:523] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:21.786751 22629 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap starting.
I20240502 21:26:21.892657 22282 raft_consensus.cc:1050] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER ee8dcd502996434ea55a16a62a5324fb to VOTER
I20240502 21:26:21.893499 22282 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:21.896672 21883 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 21:26:21.898312 22593 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 21:26:21.909163 22423 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index 66 to 71, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.915938 20696 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 21:26:21.915648 21883 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 66 to 71, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.917033 22281 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 21:26:21.921828 20697 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 66 to 71, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.925896 22644 raft_consensus.cc:1050] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b: attempting to promote NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 to VOTER
I20240502 21:26:21.926973 22644 consensus_queue.cc:237] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:21.927951 20281 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb reported cstate change: config changed from index 66 to 71, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:21.957650 22013 raft_consensus.cc:1261] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 LEARNER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:26:21.958792 22616 consensus_queue.cc:1035] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:21.958993 22647 ts_tablet_manager.cc:902] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:21.983053 20427 raft_consensus.cc:1261] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:26:21.989109 22616 consensus_queue.cc:1035] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:22.037750 22616 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.039757 22013 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.044593 20281 catalog_manager.cc:5484] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 3 to 4, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.066861 22647 tablet_copy_client.cc:308] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:22.079283 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet 9ca7cd6b72d4419aade387eaf375454e from peer 16745bde14bc4a43b532e6bb828f0625 ({username='slave'} at 127.16.122.4:41433)
I20240502 21:26:22.079512 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet 9ca7cd6b72d4419aade387eaf375454e from peer 16745bde14bc4a43b532e6bb828f0625 at {username='slave'} at 127.16.122.4:41433: session id = 16745bde14bc4a43b532e6bb828f0625-9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:22.082374 20585 tablet_copy_source_session.cc:205] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:22.083683 20427 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.087788 22629 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=692 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:22.088366 22629 tablet_bootstrap.cc:492] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: Bootstrap complete.
I20240502 21:26:22.088697 22629 ts_tablet_manager.cc:1367] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: Time spent bootstrapping tablet: real 0.303s user 0.116s sys 0.004s
I20240502 21:26:22.089248 22147 consensus_queue.cc:237] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:22.091012 22629 raft_consensus.cc:348] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:22.091267 22629 raft_consensus.cc:729] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c0aa2a3f325429eb83423bff3a80761, State: Initialized, Role: LEARNER
I20240502 21:26:22.091511 22629 consensus_queue.cc:260] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: true } }
I20240502 21:26:22.093163 22013 raft_consensus.cc:1261] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:22.094033 22651 consensus_queue.cc:1035] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:22.094573 22647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca7cd6b72d4419aade387eaf375454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.098686 22629 ts_tablet_manager.cc:1398] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:26:22.101769 22647 tablet_copy_client.cc:783] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:22.102048 22647 tablet_copy_client.cc:647] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:22.102969 22651 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.106375 22014 raft_consensus.cc:2932] T 0e14655d22984ef3aab11be93eaaa07d P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.111567 20271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e14655d22984ef3aab11be93eaaa07d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:22.113647 20281 catalog_manager.cc:5484] T 0e14655d22984ef3aab11be93eaaa07d P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 4 to 5, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.116540 20584 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session 6c0aa2a3f325429eb83423bff3a80761-8bdf4a8993904b02b2dd0fe3ba0f9e4f received from {username='slave'} at 127.16.122.6:56377
I20240502 21:26:22.116698 20584 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session 6c0aa2a3f325429eb83423bff3a80761-8bdf4a8993904b02b2dd0fe3ba0f9e4f on tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f with peer 6c0aa2a3f325429eb83423bff3a80761
I20240502 21:26:22.117816 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e14655d22984ef3aab11be93eaaa07d with delete_type TABLET_DATA_TOMBSTONED (TS 35812e2216cb4758aee1e94877a823b7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34202
I20240502 21:26:22.131788 22665 tablet_replica.cc:330] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: stopping tablet replica
I20240502 21:26:22.132189 22665 raft_consensus.cc:2229] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:22.132571 22665 raft_consensus.cc:2258] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:22.136179 22665 ts_tablet_manager.cc:1875] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:22.139935 22647 tablet_copy_client.cc:523] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:22.143226 22665 ts_tablet_manager.cc:1888] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 21:26:22.143479 22665 log.cc:1196] T 0e14655d22984ef3aab11be93eaaa07d P 35812e2216cb4758aee1e94877a823b7: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e14655d22984ef3aab11be93eaaa07d
I20240502 21:26:22.143714 22647 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: Bootstrap starting.
I20240502 21:26:22.144412 20274 catalog_manager.cc:4830] TS 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): tablet 0e14655d22984ef3aab11be93eaaa07d (table TestTable [id=4beabd6b02404fba811a489d4466be1a]) successfully deleted
I20240502 21:26:22.170192 22013 raft_consensus.cc:1203] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.57->[1.58-1.70] Dedup: 1.61->[1.62-1.70]
I20240502 21:26:22.315335 22285 raft_consensus.cc:1050] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER 6c0aa2a3f325429eb83423bff3a80761 to VOTER
I20240502 21:26:22.317559 22285 consensus_queue.cc:237] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:22.320449 22013 raft_consensus.cc:1261] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 21:26:22.321470 22593 consensus_queue.cc:1035] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240502 21:26:22.325702 22282 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 58 to 72, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.330857 22013 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 58 to 72, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.335551 20696 raft_consensus.cc:1261] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 21:26:22.339394 20280 catalog_manager.cc:5484] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 58 to 72, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.342396 22602 consensus_queue.cc:1035] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240502 21:26:22.348125 20562 consensus_queue.cc:237] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:22.351271 22013 raft_consensus.cc:1261] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240502 21:26:22.351153 20696 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 58 to 72, 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.352514 22282 consensus_queue.cc:1035] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 21:26:22.355589 22602 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 72 to 73, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.360641 22013 raft_consensus.cc:2932] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 72 to 73, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:22.361313 20272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:22.366832 20281 catalog_manager.cc:5484] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 72 to 73, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.374191 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f with delete_type TABLET_DATA_TOMBSTONED (TS 0ac8c8b961dc4500bc8b9f1055298244 not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:50304
I20240502 21:26:22.400544 20564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "380fd60c78334d90b26c3c9ce4945247"
dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53110
I20240502 21:26:22.400852 20564 raft_consensus.cc:595] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:22.403599 22593 raft_consensus.cc:979] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: : Instructing follower 16745bde14bc4a43b532e6bb828f0625 to start an election
I20240502 21:26:22.403767 22593 raft_consensus.cc:1067] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Signalling peer 16745bde14bc4a43b532e6bb828f0625 to start an election
I20240502 21:26:22.405023 21752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "380fd60c78334d90b26c3c9ce4945247"
dest_uuid: "16745bde14bc4a43b532e6bb828f0625"
from {username='slave'} at 127.16.122.2:51523
I20240502 21:26:22.405356 21752 raft_consensus.cc:482] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:22.405591 21752 raft_consensus.cc:3037] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.409744 21752 raft_consensus.cc:504] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:22.415467 22677 tablet_replica.cc:330] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: stopping tablet replica
I20240502 21:26:22.423139 21753 raft_consensus.cc:1226] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Rejecting Update request from peer 654f8d3e16bb4a658607a5c26f261cbf for earlier term 1. Current term is 2. Ops: []
I20240502 21:26:22.424640 22602 consensus_queue.cc:1046] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.78, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240502 21:26:22.425181 22602 raft_consensus.cc:3032] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:22.425477 22602 raft_consensus.cc:729] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:22.425967 22602 consensus_queue.cc:260] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:22.426718 22602 raft_consensus.cc:3037] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.445079 21752 leader_election.cc:288] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [CANDIDATE]: Term 2 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539), 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:22.447245 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fd60c78334d90b26c3c9ce4945247" candidate_uuid: "16745bde14bc4a43b532e6bb828f0625" candidate_term: 2 candidate_status { last_received { term: 1 index: 78 } } ignore_live_leader: true dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:22.451109 20564 raft_consensus.cc:2454] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16745bde14bc4a43b532e6bb828f0625 in term 2.
I20240502 21:26:22.452001 21687 leader_election.cc:302] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16745bde14bc4a43b532e6bb828f0625, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:22.455713 22677 raft_consensus.cc:2229] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:22.457396 22677 raft_consensus.cc:2258] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:22.460779 22687 raft_consensus.cc:2781] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:22.469051 22677 ts_tablet_manager.cc:1875] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:26:22.479998 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Illegal state: replica 654f8d3e16bb4a658607a5c26f261cbf is not leader of this config: current role FOLLOWER
I20240502 21:26:22.496728 22687 raft_consensus.cc:686] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 2 LEADER]: Becoming Leader. State: Replica: 16745bde14bc4a43b532e6bb828f0625, State: Running, Role: LEADER
I20240502 21:26:22.510486 22647 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=87 ignored=0} inserts{seen=718 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:22.511010 22647 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: Bootstrap complete.
I20240502 21:26:22.504979 22687 consensus_queue.cc:237] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 78, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:22.523964 22647 ts_tablet_manager.cc:1367] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: Time spent bootstrapping tablet: real 0.380s user 0.146s sys 0.012s
I20240502 21:26:22.524799 22647 raft_consensus.cc:348] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:22.525806 22647 raft_consensus.cc:729] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16745bde14bc4a43b532e6bb828f0625, State: Initialized, Role: LEARNER
I20240502 21:26:22.526033 22647 consensus_queue.cc:260] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: true } }
I20240502 21:26:22.526614 22647 ts_tablet_manager.cc:1398] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:22.528097 20585 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session 16745bde14bc4a43b532e6bb828f0625-9ca7cd6b72d4419aade387eaf375454e received from {username='slave'} at 127.16.122.4:41433
I20240502 21:26:22.528244 20585 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session 16745bde14bc4a43b532e6bb828f0625-9ca7cd6b72d4419aade387eaf375454e on tablet 9ca7cd6b72d4419aade387eaf375454e with peer 16745bde14bc4a43b532e6bb828f0625
I20240502 21:26:22.528164 20426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fd60c78334d90b26c3c9ce4945247" candidate_uuid: "16745bde14bc4a43b532e6bb828f0625" candidate_term: 2 candidate_status { last_received { term: 1 index: 78 } } ignore_live_leader: true dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:22.528440 20426 raft_consensus.cc:3037] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.532855 20426 raft_consensus.cc:2454] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16745bde14bc4a43b532e6bb828f0625 in term 2.
I20240502 21:26:22.555159 20280 catalog_manager.cc:5484] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: term changed from 1 to 2, leader changed from 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) to 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4). New cstate: current_term: 2 leader_uuid: "16745bde14bc4a43b532e6bb828f0625" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.558913 22677 ts_tablet_manager.cc:1888] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72
I20240502 21:26:22.559069 22677 log.cc:1196] T 8bdf4a8993904b02b2dd0fe3ba0f9e4f P 0ac8c8b961dc4500bc8b9f1055298244: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/8bdf4a8993904b02b2dd0fe3ba0f9e4f
I20240502 21:26:22.566814 20273 catalog_manager.cc:4830] TS 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f (table rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]) successfully deleted
W20240502 21:26:22.623739 20388 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38928: Illegal state: replica 35812e2216cb4758aee1e94877a823b7 is not leader of this config: current role FOLLOWER
I20240502 21:26:22.635051 20696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d15373bff59348b9a65b42053e8df389"
dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52012
I20240502 21:26:22.635363 20696 raft_consensus.cc:595] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:22.637414 22615 raft_consensus.cc:979] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: : Instructing follower b4edc0e5b5c04ac5a7fe3f99d86d314b to start an election
I20240502 21:26:22.637622 22615 raft_consensus.cc:1067] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Signalling peer b4edc0e5b5c04ac5a7fe3f99d86d314b to start an election
I20240502 21:26:22.639358 22148 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d15373bff59348b9a65b42053e8df389"
dest_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b"
from {username='slave'} at 127.16.122.3:59675
I20240502 21:26:22.639678 22148 raft_consensus.cc:482] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:22.639802 22148 raft_consensus.cc:3037] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.650375 22148 raft_consensus.cc:504] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:22.651573 22147 raft_consensus.cc:1226] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Rejecting Update request from peer 0ac8c8b961dc4500bc8b9f1055298244 for earlier term 1. Current term is 2. Ops: []
I20240502 21:26:22.654246 22063 consensus_queue.cc:1046] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.82, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 21:26:22.654600 22063 raft_consensus.cc:3032] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:22.654706 22063 raft_consensus.cc:729] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:22.655027 22063 consensus_queue.cc:260] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 82, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:22.655378 22063 raft_consensus.cc:3037] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.667042 20696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15373bff59348b9a65b42053e8df389" candidate_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" candidate_term: 2 candidate_status { last_received { term: 1 index: 82 } } ignore_live_leader: true dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:22.668905 22148 leader_election.cc:288] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 2 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769), 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:22.670089 20696 raft_consensus.cc:2454] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4edc0e5b5c04ac5a7fe3f99d86d314b in term 2.
I20240502 21:26:22.674027 22081 leader_election.cc:302] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, b4edc0e5b5c04ac5a7fe3f99d86d314b; no voters:
I20240502 21:26:22.674446 22651 raft_consensus.cc:2781] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Leader election won for term 2
W20240502 21:26:22.707005 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50340: Illegal state: replica 0ac8c8b961dc4500bc8b9f1055298244 is not leader of this config: current role FOLLOWER
I20240502 21:26:22.708412 20564 raft_consensus.cc:1261] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer 16745bde14bc4a43b532e6bb828f0625: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 2 index: 80. (index mismatch)
I20240502 21:26:22.709067 22689 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
W20240502 21:26:22.711753 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34290: Illegal state: replica 654f8d3e16bb4a658607a5c26f261cbf is not leader of this config: current role FOLLOWER
I20240502 21:26:22.716951 20427 raft_consensus.cc:1261] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer 16745bde14bc4a43b532e6bb828f0625: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 2 index: 80. (index mismatch)
I20240502 21:26:22.723958 22511 mvcc.cc:205] Tried to move back new op lower bound from 7023350508366766080 to 7023350507724328960. Current Snapshot: MvccSnapshot[applied={T|T < 7023350508366766080 or (T in {7023350508366766080})}]
W20240502 21:26:22.726639 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50340: Illegal state: replica 0ac8c8b961dc4500bc8b9f1055298244 is not leader of this config: current role FOLLOWER
W20240502 21:26:22.732012 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34290: Illegal state: replica 654f8d3e16bb4a658607a5c26f261cbf is not leader of this config: current role FOLLOWER
I20240502 21:26:22.733922 22651 raft_consensus.cc:686] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Becoming Leader. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Running, Role: LEADER
I20240502 21:26:22.743714 22689 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240502 21:26:22.759783 22314 mvcc.cc:205] Tried to move back new op lower bound from 7023350508366766080 to 7023350507724328960. Current Snapshot: MvccSnapshot[applied={T|T < 7023350508366766080}]
I20240502 21:26:22.771195 22651 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.82, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:22.802779 20280 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: term changed from 1 to 2, leader changed from 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) to b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7). New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.808398 21753 consensus_queue.cc:237] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 2.82, Last appended by leader: 78, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:22.813036 20427 raft_consensus.cc:1261] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer 16745bde14bc4a43b532e6bb828f0625: Log matching property violated. Preceding OpId in replica: term: 2 index: 82. Preceding OpId from leader: term: 2 index: 83. (index mismatch)
I20240502 21:26:22.815836 22689 consensus_queue.cc:1035] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 21:26:22.820775 20696 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 2 index: 84. (index mismatch)
I20240502 21:26:22.822304 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15373bff59348b9a65b42053e8df389" candidate_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" candidate_term: 2 candidate_status { last_received { term: 1 index: 82 } } ignore_live_leader: true dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:22.822707 20564 raft_consensus.cc:3037] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:22.826635 20564 raft_consensus.cc:2454] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4edc0e5b5c04ac5a7fe3f99d86d314b in term 2.
I20240502 21:26:22.827380 22710 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 [term 2 LEADER]: Committing config change with OpId 2.83: config changed from index 70 to 83, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:22.829360 20427 raft_consensus.cc:2932] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Committing config change with OpId 2.83: config changed from index 70 to 83, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:22.830772 20562 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 2 index: 84. (index mismatch)
I20240502 21:26:22.840952 20280 catalog_manager.cc:5484] T 380fd60c78334d90b26c3c9ce4945247 P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index 70 to 83, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "16745bde14bc4a43b532e6bb828f0625" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:22.845886 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet 380fd60c78334d90b26c3c9ce4945247 with delete_type TABLET_DATA_TOMBSTONED (TS 654f8d3e16bb4a658607a5c26f261cbf not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:34210
I20240502 21:26:22.856979 20272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 380fd60c78334d90b26c3c9ce4945247 with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:22.826529 22704 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.032s
I20240502 21:26:22.858983 22651 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240502 21:26:22.874673 22713 tablet_replica.cc:330] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:22.875334 22713 raft_consensus.cc:2229] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:22.875604 22713 pending_rounds.cc:62] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Trying to abort 1 pending ops.
I20240502 21:26:22.875769 22713 pending_rounds.cc:68] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Aborting op as it isn't in flight: id { term: 2 index: 82 } timestamp: 7023350508748062720 op_type: WRITE_OP write_request { tablet_id: "380fd60c78334d90b26c3c9ce4945247" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000k\004\030]N6M\002\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""b73fa62ee76ea657" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350508702498816 authz_token { token_data: "\010\243\214\320\261\006\"3\n\005slave\022*\n 3ea13f305358431fbbafab8ba82d063f\020\001\030\001 \001(\001" signature: "<redacted>""_\301v(\021\257\025\317mag\014!\020\342\2050K\357z\272Z=k\263/\330x\344\365\023\341\022d\2372\236\014\232\2301\263\356\300\262\314$7\002}\326\366\334\035\330W\264t)I\347\205R\257" signing_key_seq_num: 0 } } request_id { client_id: "ac5f24ad4a754dcfa6249d488208ef77" seq_no: 216 first_incomplete_seq_no: 216 attempt_no: 0 }
I20240502 21:26:22.876392 22552 mvcc.cc:205] Tried to move back new op lower bound from 7023350508824207360 to 7023350508666875904. Current Snapshot: MvccSnapshot[applied={T|T < 7023350508824207360 or (T in {7023350508824207360})}]
I20240502 21:26:22.876943 22713 raft_consensus.cc:2258] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:22.878062 22713 ts_tablet_manager.cc:1875] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:22.879999 22412 mvcc.cc:205] Tried to move back new op lower bound from 7023350508824207360 to 7023350508666875904. Current Snapshot: MvccSnapshot[applied={T|T < 7023350508824207360}]
I20240502 21:26:22.889776 22498 mvcc.cc:205] Tried to move back new op lower bound from 7023350508824207360 to 7023350508666875904. Current Snapshot: MvccSnapshot[applied={T|T < 7023350508824207360}]
I20240502 21:26:22.897020 22713 ts_tablet_manager.cc:1888] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.82
I20240502 21:26:22.897255 22713 log.cc:1196] T 380fd60c78334d90b26c3c9ce4945247 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/380fd60c78334d90b26c3c9ce4945247
I20240502 21:26:22.902400 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet 380fd60c78334d90b26c3c9ce4945247 (table rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]) successfully deleted
I20240502 21:26:22.906625 22148 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 2.84, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:22.910878 20564 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 84. Preceding OpId from leader: term: 2 index: 85. (index mismatch)
I20240502 21:26:22.911464 22704 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 21:26:22.914534 22704 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.85: config changed from index 70 to 85, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:22.920917 20562 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Committing config change with OpId 2.85: config changed from index 70 to 85, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:22.922655 20271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d15373bff59348b9a65b42053e8df389 with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:22.930997 20280 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 70 to 85, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.947345 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet d15373bff59348b9a65b42053e8df389 with delete_type TABLET_DATA_TOMBSTONED (TS 0ac8c8b961dc4500bc8b9f1055298244 not found in new config with opid_index 85) from {username='slave'} at 127.0.0.1:50304
I20240502 21:26:22.947813 22677 tablet_replica.cc:330] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: stopping tablet replica
I20240502 21:26:22.948040 22677 raft_consensus.cc:2229] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:22.948279 22677 raft_consensus.cc:2258] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:22.949487 22677 ts_tablet_manager.cc:1875] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:22.958861 22677 ts_tablet_manager.cc:1888] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.84
I20240502 21:26:22.959707 22677 log.cc:1196] T d15373bff59348b9a65b42053e8df389 P 0ac8c8b961dc4500bc8b9f1055298244: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/d15373bff59348b9a65b42053e8df389
I20240502 21:26:22.978708 20273 catalog_manager.cc:4830] TS 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): tablet d15373bff59348b9a65b42053e8df389 (table rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]) successfully deleted
I20240502 21:26:23.009976 21752 raft_consensus.cc:1203] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.75->[1.76-1.103] Dedup: 1.88->[1.89-1.103]
I20240502 21:26:23.164988 22281 raft_consensus.cc:1050] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER 16745bde14bc4a43b532e6bb828f0625 to VOTER
I20240502 21:26:23.165810 22281 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:23.176148 21752 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 21:26:23.176924 22602 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 21:26:23.180200 22423 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 76 to 104, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:23.183854 20698 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 21:26:23.188122 21753 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 76 to 104, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:23.194736 22282 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 21:26:23.198818 20696 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 76 to 104, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:23.209779 20280 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 76 to 104, 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:23.742331 20696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2998322d830948ac858235f4fb7defca"
dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52042
I20240502 21:26:23.742602 20696 raft_consensus.cc:595] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Received request to transfer leadership
W20240502 21:26:23.764264 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.785374 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.799528 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.807106 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.827476 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.839411 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.851369 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.869985 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.888667 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.909979 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.930766 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.950599 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.968207 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:23.985230 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
I20240502 21:26:23.985244 20564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0892742cff44146ae2fb345d7e2bd3a"
dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53150
I20240502 21:26:23.985531 20564 raft_consensus.cc:595] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:23.986433 22282 raft_consensus.cc:979] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: : Instructing follower 35812e2216cb4758aee1e94877a823b7 to start an election
I20240502 21:26:23.986613 22281 raft_consensus.cc:1067] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Signalling peer 35812e2216cb4758aee1e94877a823b7 to start an election
I20240502 21:26:23.987718 20426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0892742cff44146ae2fb345d7e2bd3a"
dest_uuid: "35812e2216cb4758aee1e94877a823b7"
from {username='slave'} at 127.16.122.2:36075
I20240502 21:26:23.987993 20426 raft_consensus.cc:482] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:23.988147 20426 raft_consensus.cc:3037] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:23.991741 20426 raft_consensus.cc:504] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:23.994258 20426 leader_election.cc:288] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149), 6c0aa2a3f325429eb83423bff3a80761 (127.16.122.6:38855)
I20240502 21:26:24.003819 22014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0892742cff44146ae2fb345d7e2bd3a" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 127 } } ignore_live_leader: true dest_uuid: "6c0aa2a3f325429eb83423bff3a80761"
I20240502 21:26:24.004227 22014 raft_consensus.cc:3037] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:24.004491 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0892742cff44146ae2fb345d7e2bd3a" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 127 } } ignore_live_leader: true dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:24.004715 20564 raft_consensus.cc:3032] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:24.004794 20564 raft_consensus.cc:729] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:24.005116 20564 consensus_queue.cc:260] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.127, Last appended by leader: 127, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:24.005515 20564 raft_consensus.cc:3037] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:24.007846 22014 raft_consensus.cc:2454] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 2.
I20240502 21:26:24.008266 20564 raft_consensus.cc:2454] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 2.
I20240502 21:26:24.008497 20365 leader_election.cc:302] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35812e2216cb4758aee1e94877a823b7, 6c0aa2a3f325429eb83423bff3a80761; no voters:
I20240502 21:26:24.008910 22682 raft_consensus.cc:2781] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:24.009064 22682 raft_consensus.cc:686] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 2 LEADER]: Becoming Leader. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Running, Role: LEADER
I20240502 21:26:24.009423 22682 consensus_queue.cc:237] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.127, Last appended by leader: 127, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
W20240502 21:26:24.011816 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
I20240502 21:26:24.012843 20281 catalog_manager.cc:5484] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 reported cstate change: term changed from 1 to 2, leader changed from 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) to 35812e2216cb4758aee1e94877a823b7 (127.16.122.1). New cstate: current_term: 2 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:24.016185 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Illegal state: replica 654f8d3e16bb4a658607a5c26f261cbf is not leader of this config: current role FOLLOWER
W20240502 21:26:24.036813 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:24.058645 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:24.085115 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:24.116451 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:24.140425 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
I20240502 21:26:24.151010 20563 raft_consensus.cc:1261] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 2 index: 129. (index mismatch)
I20240502 21:26:24.151407 22013 raft_consensus.cc:1261] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 2 index: 129. (index mismatch)
I20240502 21:26:24.152388 22676 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20240502 21:26:24.152901 22682 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
W20240502 21:26:24.166769 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
W20240502 21:26:24.195500 20659 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
I20240502 21:26:24.202998 20427 consensus_queue.cc:237] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.130, Last appended by leader: 127, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } }
I20240502 21:26:24.208981 22013 raft_consensus.cc:1261] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 130. Preceding OpId from leader: term: 2 index: 131. (index mismatch)
I20240502 21:26:24.210479 22676 consensus_queue.cc:1035] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:24.215546 22787 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 [term 2 LEADER]: Committing config change with OpId 2.131: config changed from index 94 to 131, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:24.219628 20274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0892742cff44146ae2fb345d7e2bd3a with cas_config_opid_index 94: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:24.220793 22013 raft_consensus.cc:2932] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Committing config change with OpId 2.131: config changed from index 94 to 131, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } } }
I20240502 21:26:24.224123 22223 raft_consensus.cc:979] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: : Instructing follower ee8dcd502996434ea55a16a62a5324fb to start an election
I20240502 21:26:24.224359 22223 raft_consensus.cc:1067] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Signalling peer ee8dcd502996434ea55a16a62a5324fb to start an election
I20240502 21:26:24.225390 21884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2998322d830948ac858235f4fb7defca"
dest_uuid: "ee8dcd502996434ea55a16a62a5324fb"
from {username='slave'} at 127.16.122.3:37035
I20240502 21:26:24.225801 21884 raft_consensus.cc:482] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:24.226017 21884 raft_consensus.cc:3037] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:26:24.227511 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Service unavailable: leader transfer in progress
I20240502 21:26:24.231927 21884 raft_consensus.cc:504] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:24.236622 21884 leader_election.cc:288] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [CANDIDATE]: Term 2 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539), 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:24.251991 20563 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e"
dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53162
I20240502 21:26:24.252256 20563 raft_consensus.cc:595] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Received request to transfer leadership
W20240502 21:26:24.252862 22412 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.254034 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998322d830948ac858235f4fb7defca" candidate_uuid: "ee8dcd502996434ea55a16a62a5324fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 117 } } ignore_live_leader: true dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:24.254298 20698 raft_consensus.cc:3032] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:24.254437 20698 raft_consensus.cc:729] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:24.254762 20698 consensus_queue.cc:260] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.117, Last appended by leader: 117, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:24.249989 20281 catalog_manager.cc:5484] T d0892742cff44146ae2fb345d7e2bd3a P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index 94 to 131, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0aa2a3f325429eb83423bff3a80761" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 38855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:26:24.261087 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.262923 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet d0892742cff44146ae2fb345d7e2bd3a with delete_type TABLET_DATA_TOMBSTONED (TS 654f8d3e16bb4a658607a5c26f261cbf not found in new config with opid_index 131) from {username='slave'} at 127.0.0.1:34210
I20240502 21:26:24.263680 22796 tablet_replica.cc:330] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:24.263993 22796 raft_consensus.cc:2229] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:24.264289 22796 pending_rounds.cc:62] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Trying to abort 1 pending ops.
I20240502 21:26:24.264586 22796 pending_rounds.cc:68] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Aborting op as it isn't in flight: id { term: 2 index: 130 } timestamp: 7023350514463752192 op_type: WRITE_OP write_request { tablet_id: "d0892742cff44146ae2fb345d7e2bd3a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000<-a\177C\204\244\217\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""e0f437ee0307f652" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350514421350400 authz_token { token_data: "\010\243\214\320\261\006\"3\n\005slave\022*\n b813afcdc0bb448499de47644d610fac\020\001\030\001 \001(\001" signature: "<redacted>""\013\277\003mv\377\215<\023\017\274\322\206\205\270\271\255 !\013\256N\025}aN\341\225\366\024\n\351q7\335\246\020\321E\215\300\2162.\226\222T^\277(R}\0306\253\0371L\023\354\377\332\232\220" signing_key_seq_num: 0 } } request_id { client_id: "3900c5d1122e4f78a18eac99a9cebc91" seq_no: 350 first_incomplete_seq_no: 350 attempt_no: 0 }
W20240502 21:26:24.267844 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50356: Illegal state: replica 0ac8c8b961dc4500bc8b9f1055298244 is not leader of this config: current role FOLLOWER
I20240502 21:26:24.268569 20698 raft_consensus.cc:3037] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:26:24.267516 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.279430 20698 raft_consensus.cc:2454] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8dcd502996434ea55a16a62a5324fb in term 2.
I20240502 21:26:24.280598 21819 leader_election.cc:302] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, ee8dcd502996434ea55a16a62a5324fb; no voters:
I20240502 21:26:24.281049 22796 raft_consensus.cc:2258] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:24.282050 22796 ts_tablet_manager.cc:1875] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:26:24.286029 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.296278 20388 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40624: Illegal state: replica 35812e2216cb4758aee1e94877a823b7 is not leader of this config: current role FOLLOWER
W20240502 21:26:24.305146 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.307906 22798 raft_consensus.cc:2781] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Leader election won for term 2
W20240502 21:26:24.317625 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.322852 22798 raft_consensus.cc:686] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Becoming Leader. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Running, Role: LEADER
I20240502 21:26:24.323493 22798 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.117, Last appended by leader: 117, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:24.325534 20426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998322d830948ac858235f4fb7defca" candidate_uuid: "ee8dcd502996434ea55a16a62a5324fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 117 } } ignore_live_leader: true dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:24.325784 20426 raft_consensus.cc:3037] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:24.328032 20281 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb reported cstate change: term changed from 1 to 2, leader changed from 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) to ee8dcd502996434ea55a16a62a5324fb (127.16.122.5). New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:24.328696 20426 raft_consensus.cc:2454] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8dcd502996434ea55a16a62a5324fb in term 2.
W20240502 21:26:24.332350 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.345595 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.361238 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.373083 22796 ts_tablet_manager.cc:1888] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.130
I20240502 21:26:24.373276 22796 log.cc:1196] T d0892742cff44146ae2fb345d7e2bd3a P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/d0892742cff44146ae2fb345d7e2bd3a
W20240502 21:26:24.375679 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.376901 22205 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.16.122.62:40689, user_credentials={real_user=slave}} blocked reactor thread for 44233.4us
I20240502 21:26:24.391409 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet d0892742cff44146ae2fb345d7e2bd3a (table rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]) successfully deleted
W20240502 21:26:24.393868 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.414320 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.436156 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.453086 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.478547 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.501938 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.514566 20698 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 2 index: 119. (index mismatch)
I20240502 21:26:24.514990 20426 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 2 index: 119. (index mismatch)
I20240502 21:26:24.516007 22798 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 21:26:24.516973 22798 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 21:26:24.524389 22595 mvcc.cc:205] Tried to move back new op lower bound from 7023350515763310592 to 7023350514994237440. Current Snapshot: MvccSnapshot[applied={T|T < 7023350515763310592 or (T in {7023350515763310592})}]
W20240502 21:26:24.525182 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.550799 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
W20240502 21:26:24.574508 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Service unavailable: leader transfer in progress
I20240502 21:26:24.579155 22734 raft_consensus.cc:979] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: : Instructing follower 0ac8c8b961dc4500bc8b9f1055298244 to start an election
I20240502 21:26:24.579326 22734 raft_consensus.cc:1067] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Signalling peer 0ac8c8b961dc4500bc8b9f1055298244 to start an election
I20240502 21:26:24.580286 20698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e"
dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
from {username='slave'} at 127.16.122.2:46387
I20240502 21:26:24.580528 20698 raft_consensus.cc:482] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:24.580655 20698 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:24.584360 20698 raft_consensus.cc:504] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:24.585677 20563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:24.592803 20698 leader_election.cc:288] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 2 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149), 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571)
I20240502 21:26:24.585968 20563 raft_consensus.cc:3032] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:24.593150 20563 raft_consensus.cc:729] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:24.598110 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e" candidate_uuid: "0ac8c8b961dc4500bc8b9f1055298244" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "16745bde14bc4a43b532e6bb828f0625"
I20240502 21:26:24.598498 21751 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:26:24.600432 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34280: Illegal state: replica 654f8d3e16bb4a658607a5c26f261cbf is not leader of this config: current role FOLLOWER
I20240502 21:26:24.603497 21751 raft_consensus.cc:2454] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 2.
I20240502 21:26:24.604744 20632 leader_election.cc:302] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 16745bde14bc4a43b532e6bb828f0625; no voters:
I20240502 21:26:24.605085 22615 raft_consensus.cc:2781] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:24.605284 22615 raft_consensus.cc:686] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Becoming Leader. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:24.605544 22615 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 122, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:24.609725 20280 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: term changed from 1 to 2, leader changed from 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) to 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3). New cstate: current_term: 2 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:24.603873 20563 consensus_queue.cc:260] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 122, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:24.624269 20563 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:24.632534 20563 raft_consensus.cc:2454] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac8c8b961dc4500bc8b9f1055298244 in term 2.
I20240502 21:26:24.635295 21884 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 2.123, Last appended by leader: 117, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:24.639724 20427 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 123. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 21:26:24.640641 22812 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 21:26:24.647176 22798 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.124: config changed from index 84 to 124, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:24.661895 20280 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb reported cstate change: config changed from index 84 to 124, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:24.667785 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet 2998322d830948ac858235f4fb7defca with delete_type TABLET_DATA_TOMBSTONED (TS 0ac8c8b961dc4500bc8b9f1055298244 not found in new config with opid_index 124) from {username='slave'} at 127.0.0.1:50304
I20240502 21:26:24.671708 20426 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Committing config change with OpId 2.124: config changed from index 84 to 124, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:24.700853 20274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2998322d830948ac858235f4fb7defca with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:24.715088 22817 tablet_replica.cc:330] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: stopping tablet replica
I20240502 21:26:24.715427 22817 raft_consensus.cc:2229] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:24.715879 22817 pending_rounds.cc:62] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Trying to abort 1 pending ops.
I20240502 21:26:24.715996 22817 pending_rounds.cc:68] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Aborting op as it isn't in flight: id { term: 2 index: 123 } timestamp: 7023350516166778880 op_type: WRITE_OP write_request { tablet_id: "2998322d830948ac858235f4fb7defca" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\003\017\370_\235p\016U\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6b44df97ed66d30b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350516041785344 authz_token { token_data: "\010\243\214\320\261\006\"3\n\005slave\022*\n 518f4738239746d6b8d9a481360bac0d\020\001\030\001 \001(\001" signature: "<redacted>""j\331\260FY\265\"@\365\251\320\'\241n_\302\035\330\200\246\035\315U\010O\264\341\234\366t\364L\352\251\276\266~\306\031,@\307\234|@m\302\366\214\237\240\217\233\362\3565\245\032U\262\214\366\301\350" signing_key_seq_num: 0 } } request_id { client_id: "03f043aa09bb486ba32ad7dcdab92ab0" seq_no: 374 first_incomplete_seq_no: 374 attempt_no: 0 }
I20240502 21:26:24.716714 22817 raft_consensus.cc:2258] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:24.719736 22817 ts_tablet_manager.cc:1875] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:24.725689 22817 ts_tablet_manager.cc:1888] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.123
I20240502 21:26:24.725821 22817 log.cc:1196] T 2998322d830948ac858235f4fb7defca P 0ac8c8b961dc4500bc8b9f1055298244: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/2998322d830948ac858235f4fb7defca
I20240502 21:26:24.726745 20273 catalog_manager.cc:4830] TS 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): tablet 2998322d830948ac858235f4fb7defca (table rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]) successfully deleted
I20240502 21:26:24.763126 20564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a"
dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53178
I20240502 21:26:24.763906 20564 raft_consensus.cc:595] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Received request to transfer leadership
W20240502 21:26:24.773242 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
I20240502 21:26:24.780062 20564 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 21:26:24.780087 21753 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
W20240502 21:26:24.781824 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
I20240502 21:26:24.782624 22615 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 21:26:24.782927 22223 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 21:26:24.793537 20696 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 2.124, Last appended by leader: 122, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } }
I20240502 21:26:24.795365 21752 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 2 index: 124. Preceding OpId from leader: term: 2 index: 125. (index mismatch)
I20240502 21:26:24.796092 22821 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 21:26:24.802670 22615 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Committing config change with OpId 2.125: config changed from index 104 to 125, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
I20240502 21:26:24.803588 21753 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Committing config change with OpId 2.125: config changed from index 104 to 125, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
W20240502 21:26:24.806838 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
I20240502 21:26:24.807309 20273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ca7cd6b72d4419aade387eaf375454e with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:24.809599 20280 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: config changed from index 104 to 125, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } }
W20240502 21:26:24.817519 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
I20240502 21:26:24.828569 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca7cd6b72d4419aade387eaf375454e with delete_type TABLET_DATA_TOMBSTONED (TS 654f8d3e16bb4a658607a5c26f261cbf not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:34210
I20240502 21:26:24.828994 22796 tablet_replica.cc:330] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:24.829218 22796 raft_consensus.cc:2229] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:24.829453 22796 raft_consensus.cc:2258] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:24.835242 22796 ts_tablet_manager.cc:1875] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:26:24.840014 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:24.852154 20520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:24.876749 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
I20240502 21:26:24.882660 22796 ts_tablet_manager.cc:1888] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.122
I20240502 21:26:24.882835 22796 log.cc:1196] T 9ca7cd6b72d4419aade387eaf375454e P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:24.883916 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet 9ca7cd6b72d4419aade387eaf375454e (table rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]) successfully deleted
W20240502 21:26:24.889499 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:24.902375 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:24.935801 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:24.970724 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:24.984480 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:25.002751 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:25.049275 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:25.067152 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:25.091001 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
W20240502 21:26:25.114259 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Service unavailable: leader transfer in progress
I20240502 21:26:25.116698 22282 raft_consensus.cc:979] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: : Instructing follower ee8dcd502996434ea55a16a62a5324fb to start an election
I20240502 21:26:25.116887 22282 raft_consensus.cc:1067] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Signalling peer ee8dcd502996434ea55a16a62a5324fb to start an election
I20240502 21:26:25.117642 21884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a"
dest_uuid: "ee8dcd502996434ea55a16a62a5324fb"
from {username='slave'} at 127.16.122.2:36801
I20240502 21:26:25.117928 21884 raft_consensus.cc:482] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:25.118067 21884 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:25.121065 21884 raft_consensus.cc:504] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:25.123751 20563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a" candidate_uuid: "ee8dcd502996434ea55a16a62a5324fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: true dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:25.124055 20563 raft_consensus.cc:3032] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:25.124153 20563 raft_consensus.cc:729] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:25.124476 20563 consensus_queue.cc:260] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:25.124913 20563 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:25.139791 21884 leader_election.cc:288] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [CANDIDATE]: Term 2 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149), 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:25.140144 20563 raft_consensus.cc:2454] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8dcd502996434ea55a16a62a5324fb in term 2.
I20240502 21:26:25.140297 20696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a" candidate_uuid: "ee8dcd502996434ea55a16a62a5324fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: true dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:25.140547 20696 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:26:25.140784 20363 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.2:40149, user_credentials={real_user=slave}} blocked reactor thread for 116561us
I20240502 21:26:25.143110 20696 raft_consensus.cc:2454] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8dcd502996434ea55a16a62a5324fb in term 2.
I20240502 21:26:25.145150 21819 leader_election.cc:302] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, ee8dcd502996434ea55a16a62a5324fb; no voters:
I20240502 21:26:25.145481 22799 raft_consensus.cc:2781] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:25.145612 22799 raft_consensus.cc:686] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Becoming Leader. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Running, Role: LEADER
I20240502 21:26:25.145896 22799 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:25.148917 20287 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb reported cstate change: term changed from 1 to 2, leader changed from 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) to ee8dcd502996434ea55a16a62a5324fb (127.16.122.5). New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:26:25.155576 20521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34306: Illegal state: replica 654f8d3e16bb4a658607a5c26f261cbf is not leader of this config: current role FOLLOWER
W20240502 21:26:25.229714 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52068: Illegal state: replica 0ac8c8b961dc4500bc8b9f1055298244 is not leader of this config: current role FOLLOWER
I20240502 21:26:25.292325 20696 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 2 index: 109. (index mismatch)
I20240502 21:26:25.292887 20563 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 2 index: 109. (index mismatch)
I20240502 21:26:25.293483 22849 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240502 21:26:25.293999 22849 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240502 21:26:25.334314 21882 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 2.109, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:25.338743 20696 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 109. Preceding OpId from leader: term: 2 index: 110. (index mismatch)
I20240502 21:26:25.340440 22798 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240502 21:26:25.346069 22799 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.110: config changed from index 71 to 110, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:25.352068 20274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4aeebd7828643ccab4aa62d4fd4d12a with cas_config_opid_index 71: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:25.357105 20696 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Committing config change with OpId 2.110: config changed from index 71 to 110, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:25.359036 20287 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb reported cstate change: config changed from index 71 to 110, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:25.376963 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet d4aeebd7828643ccab4aa62d4fd4d12a with delete_type TABLET_DATA_TOMBSTONED (TS 654f8d3e16bb4a658607a5c26f261cbf not found in new config with opid_index 110) from {username='slave'} at 127.0.0.1:34210
I20240502 21:26:25.377369 22796 tablet_replica.cc:330] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:25.377584 22796 raft_consensus.cc:2229] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:25.377879 22796 raft_consensus.cc:2258] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:25.379063 22796 ts_tablet_manager.cc:1875] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:25.409417 22796 ts_tablet_manager.cc:1888] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.109
I20240502 21:26:25.409636 22796 log.cc:1196] T d4aeebd7828643ccab4aa62d4fd4d12a P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:25.410444 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet d4aeebd7828643ccab4aa62d4fd4d12a (table rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]) successfully deleted
I20240502 21:26:26.042580 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:26.127105 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:26.146612 20678 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:26.188686 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:26.339514 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:26.402786 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:26.437919 21864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:26.692755 20696 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 1.176, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:26.695065 21753 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240502 21:26:26.697762 22821 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 175, Time since last communication: 0.000s
W20240502 21:26:26.700465 20635 consensus_peers.cc:485] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311): Couldn't send request to peer ee8dcd502996434ea55a16a62a5324fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12887227115743afacb541c753db51c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:26.701834 22223 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.177: config changed from index 104 to 177, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:26.710366 20280 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 104 to 177, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:26.721513 21753 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 104 to 177, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:26.813268 22147 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 2.149, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:26.817809 20563 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 149. Preceding OpId from leader: term: 2 index: 150. (index mismatch)
I20240502 21:26:26.818564 22704 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240502 21:26:26.825743 22651 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.150: config changed from index 85 to 150, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:26.830926 20563 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Committing config change with OpId 2.150: config changed from index 85 to 150, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
I20240502 21:26:26.838608 20281 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 85 to 150, NON_VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) added. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } } }
W20240502 21:26:26.850029 22082 consensus_peers.cc:485] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b -> Peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311): Couldn't send request to peer ee8dcd502996434ea55a16a62a5324fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d15373bff59348b9a65b42053e8df389. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:26.919723 21884 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 178, Committed index: 178, Last appended: 2.178, Last appended by leader: 117, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:26.933182 20425 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 178. Preceding OpId from leader: term: 2 index: 179. (index mismatch)
I20240502 21:26:26.940140 22899 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240502 21:26:26.943998 22899 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.179: config changed from index 124 to 179, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
I20240502 21:26:26.945322 20427 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Committing config change with OpId 2.179: config changed from index 124 to 179, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
W20240502 21:26:26.964403 21817 consensus_peers.cc:485] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb -> Peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365): Couldn't send request to peer b4edc0e5b5c04ac5a7fe3f99d86d314b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2998322d830948ac858235f4fb7defca. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:26.969522 20281 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index 124 to 179, NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) added. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } } }
I20240502 21:26:27.119796 22958 ts_tablet_manager.cc:902] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:27.159493 22958 tablet_copy_client.cc:308] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:27.161607 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet 12887227115743afacb541c753db51c9 from peer ee8dcd502996434ea55a16a62a5324fb ({username='slave'} at 127.16.122.5:59733)
I20240502 21:26:27.161804 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet 12887227115743afacb541c753db51c9 from peer ee8dcd502996434ea55a16a62a5324fb at {username='slave'} at 127.16.122.5:59733: session id = ee8dcd502996434ea55a16a62a5324fb-12887227115743afacb541c753db51c9
I20240502 21:26:27.166805 20719 tablet_copy_source_session.cc:205] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:27.177984 22958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12887227115743afacb541c753db51c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.190042 22958 tablet_copy_client.cc:783] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:27.190626 22958 tablet_copy_client.cc:647] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:27.196420 22958 tablet_copy_client.cc:523] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:27.205650 22958 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: Bootstrap starting.
I20240502 21:26:27.222987 22974 ts_tablet_manager.cc:902] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: Initiating tablet copy from peer b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7:34365)
I20240502 21:26:27.268776 22974 tablet_copy_client.cc:308] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.7:34365
I20240502 21:26:27.269970 22168 tablet_copy_service.cc:140] P b4edc0e5b5c04ac5a7fe3f99d86d314b: Received BeginTabletCopySession request for tablet d15373bff59348b9a65b42053e8df389 from peer ee8dcd502996434ea55a16a62a5324fb ({username='slave'} at 127.16.122.5:41585)
I20240502 21:26:27.270205 22168 tablet_copy_service.cc:161] P b4edc0e5b5c04ac5a7fe3f99d86d314b: Beginning new tablet copy session on tablet d15373bff59348b9a65b42053e8df389 from peer ee8dcd502996434ea55a16a62a5324fb at {username='slave'} at 127.16.122.5:41585: session id = ee8dcd502996434ea55a16a62a5324fb-d15373bff59348b9a65b42053e8df389
I20240502 21:26:27.273284 22168 tablet_copy_source_session.cc:205] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:27.275517 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15373bff59348b9a65b42053e8df389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.282927 22974 tablet_copy_client.cc:783] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:27.283317 22974 tablet_copy_client.cc:647] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:27.289760 22974 tablet_copy_client.cc:523] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:27.300957 22974 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: Bootstrap starting.
I20240502 21:26:27.711791 22958 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=184 overwritten=0 applied=183 ignored=0} inserts{seen=865 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:27.712637 22958 tablet_bootstrap.cc:492] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: Bootstrap complete.
I20240502 21:26:27.713140 22958 ts_tablet_manager.cc:1367] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: Time spent bootstrapping tablet: real 0.508s user 0.289s sys 0.010s
I20240502 21:26:27.714294 22958 raft_consensus.cc:348] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:27.744930 22983 ts_tablet_manager.cc:902] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: Initiating tablet copy from peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311)
I20240502 21:26:27.788570 22958 raft_consensus.cc:729] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Initialized, Role: LEARNER
I20240502 21:26:27.789018 22958 consensus_queue.cc:260] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 183, Last appended: 1.184, Last appended by leader: 184, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:27.789943 22958 ts_tablet_manager.cc:1398] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb: Time spent starting tablet: real 0.077s user 0.026s sys 0.049s
I20240502 21:26:27.790045 22983 tablet_copy_client.cc:308] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.5:35311
I20240502 21:26:27.792083 21904 tablet_copy_service.cc:140] P ee8dcd502996434ea55a16a62a5324fb: Received BeginTabletCopySession request for tablet 2998322d830948ac858235f4fb7defca from peer b4edc0e5b5c04ac5a7fe3f99d86d314b ({username='slave'} at 127.16.122.7:59377)
I20240502 21:26:27.792343 21904 tablet_copy_service.cc:161] P ee8dcd502996434ea55a16a62a5324fb: Beginning new tablet copy session on tablet 2998322d830948ac858235f4fb7defca from peer b4edc0e5b5c04ac5a7fe3f99d86d314b at {username='slave'} at 127.16.122.7:59377: session id = b4edc0e5b5c04ac5a7fe3f99d86d314b-2998322d830948ac858235f4fb7defca
I20240502 21:26:27.793503 22974 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=158 overwritten=0 applied=158 ignored=0} inserts{seen=785 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:27.794049 22974 tablet_bootstrap.cc:492] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: Bootstrap complete.
I20240502 21:26:27.795246 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session ee8dcd502996434ea55a16a62a5324fb-12887227115743afacb541c753db51c9 received from {username='slave'} at 127.16.122.5:59733
I20240502 21:26:27.795406 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session ee8dcd502996434ea55a16a62a5324fb-12887227115743afacb541c753db51c9 on tablet 12887227115743afacb541c753db51c9 with peer ee8dcd502996434ea55a16a62a5324fb
I20240502 21:26:27.796759 22974 ts_tablet_manager.cc:1367] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: Time spent bootstrapping tablet: real 0.494s user 0.259s sys 0.032s
I20240502 21:26:27.800169 22974 raft_consensus.cc:348] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:27.800580 22974 raft_consensus.cc:729] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Initialized, Role: LEARNER
I20240502 21:26:27.800936 22974 consensus_queue.cc:260] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 158, Last appended: 2.158, Last appended by leader: 158, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: true } }
I20240502 21:26:27.801676 22974 ts_tablet_manager.cc:1398] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:27.812845 21904 tablet_copy_source_session.cc:205] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:27.814708 22168 tablet_copy_service.cc:342] P b4edc0e5b5c04ac5a7fe3f99d86d314b: Request end of tablet copy session ee8dcd502996434ea55a16a62a5324fb-d15373bff59348b9a65b42053e8df389 received from {username='slave'} at 127.16.122.5:41585
I20240502 21:26:27.814864 22168 tablet_copy_service.cc:433] P b4edc0e5b5c04ac5a7fe3f99d86d314b: ending tablet copy session ee8dcd502996434ea55a16a62a5324fb-d15373bff59348b9a65b42053e8df389 on tablet d15373bff59348b9a65b42053e8df389 with peer ee8dcd502996434ea55a16a62a5324fb
I20240502 21:26:27.814909 22983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998322d830948ac858235f4fb7defca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.852916 22983 tablet_copy_client.cc:783] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:27.854966 22983 tablet_copy_client.cc:647] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:27.872452 22983 tablet_copy_client.cc:523] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:27.935855 22983 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap starting.
I20240502 21:26:28.004905 21884 raft_consensus.cc:1203] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.149->[2.150-2.166] Dedup: 2.158->[2.159-2.166]
I20240502 21:26:28.038808 21883 raft_consensus.cc:1203] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.176->[1.177-1.190] Dedup: 1.184->[1.185-1.190]
I20240502 21:26:28.143772 22615 raft_consensus.cc:1050] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER ee8dcd502996434ea55a16a62a5324fb to VOTER
I20240502 21:26:28.144697 22615 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 1.194, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.149147 21883 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 21:26:28.149894 22821 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240502 21:26:28.154521 22882 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Committing config change with OpId 1.195: config changed from index 177 to 195, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.155452 21883 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 177 to 195, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.161980 20281 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 177 to 195, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:28.198976 21752 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 21:26:28.232431 22882 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240502 21:26:28.236050 21752 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 177 to 195, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.268829 22704 raft_consensus.cc:1050] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b: attempting to promote NON_VOTER ee8dcd502996434ea55a16a62a5324fb to VOTER
I20240502 21:26:28.269909 22704 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 2.167, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.272686 21883 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 LEARNER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 167. Preceding OpId from leader: term: 2 index: 168. (index mismatch)
I20240502 21:26:28.273536 22651 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240502 21:26:28.277129 22949 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.168: config changed from index 150 to 168, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.281739 20563 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 167. Preceding OpId from leader: term: 2 index: 168. (index mismatch)
I20240502 21:26:28.282884 22651 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240502 21:26:28.284795 20562 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Committing config change with OpId 2.168: config changed from index 150 to 168, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.285487 21883 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Committing config change with OpId 2.168: config changed from index 150 to 168, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.391252 20284 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 150 to 168, ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.434092 22147 consensus_queue.cc:237] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 2.171, Last appended by leader: 82, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.452952 21883 raft_consensus.cc:1261] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Refusing update from remote peer b4edc0e5b5c04ac5a7fe3f99d86d314b: Log matching property violated. Preceding OpId in replica: term: 2 index: 171. Preceding OpId from leader: term: 2 index: 172. (index mismatch)
I20240502 21:26:28.453915 22704 consensus_queue.cc:1035] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I20240502 21:26:28.459628 22704 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEADER]: Committing config change with OpId 2.172: config changed from index 168 to 172, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.466316 20696 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12887227115743afacb541c753db51c9"
dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33262
I20240502 21:26:28.466567 20696 raft_consensus.cc:595] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:28.483865 20271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d15373bff59348b9a65b42053e8df389 with cas_config_opid_index 168: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:28.487509 20278 catalog_manager.cc:5484] T d15373bff59348b9a65b42053e8df389 P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 168 to 172, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" committed_config { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:28.490067 22223 raft_consensus.cc:979] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: : Instructing follower ee8dcd502996434ea55a16a62a5324fb to start an election
I20240502 21:26:28.489868 21883 raft_consensus.cc:2932] T d15373bff59348b9a65b42053e8df389 P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Committing config change with OpId 2.172: config changed from index 168 to 172, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.490258 22223 raft_consensus.cc:1067] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Signalling peer ee8dcd502996434ea55a16a62a5324fb to start an election
W20240502 21:26:28.503904 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50372: Service unavailable: leader transfer in progress
I20240502 21:26:28.512737 21883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12887227115743afacb541c753db51c9"
dest_uuid: "ee8dcd502996434ea55a16a62a5324fb"
from {username='slave'} at 127.16.122.3:37035
I20240502 21:26:28.513217 21883 raft_consensus.cc:482] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:28.513509 21883 raft_consensus.cc:3037] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:28.515221 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet d15373bff59348b9a65b42053e8df389 with delete_type TABLET_DATA_TOMBSTONED (TS 654f8d3e16bb4a658607a5c26f261cbf not found in new config with opid_index 172) from {username='slave'} at 127.0.0.1:34210
W20240502 21:26:28.521382 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50372: Service unavailable: leader transfer in progress
I20240502 21:26:28.520843 21883 raft_consensus.cc:504] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.523281 20696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12887227115743afacb541c753db51c9" candidate_uuid: "ee8dcd502996434ea55a16a62a5324fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: true dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:28.523504 20696 raft_consensus.cc:3032] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:28.523617 20696 raft_consensus.cc:729] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:28.523895 20696 consensus_queue.cc:260] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 202, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.524284 20696 raft_consensus.cc:3037] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:26:28.528942 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50372: Illegal state: replica 0ac8c8b961dc4500bc8b9f1055298244 is not leader of this config: current role FOLLOWER
I20240502 21:26:28.547693 23033 tablet_replica.cc:330] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:28.547952 23033 raft_consensus.cc:2229] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus shutting down.
W20240502 21:26:28.548280 20657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50372: Illegal state: replica 0ac8c8b961dc4500bc8b9f1055298244 is not leader of this config: current role FOLLOWER
I20240502 21:26:28.549929 23033 raft_consensus.cc:2258] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:28.551235 23033 ts_tablet_manager.cc:1875] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:28.578927 20696 raft_consensus.cc:2454] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8dcd502996434ea55a16a62a5324fb in term 2.
I20240502 21:26:28.579886 21819 leader_election.cc:302] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, ee8dcd502996434ea55a16a62a5324fb; no voters:
I20240502 21:26:28.580454 22849 raft_consensus.cc:2781] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:28.580615 22849 raft_consensus.cc:686] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Becoming Leader. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Running, Role: LEADER
I20240502 21:26:28.580938 22849 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 202, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.589812 21883 leader_election.cc:288] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [CANDIDATE]: Term 2 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769), 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4:46571)
I20240502 21:26:28.600796 20287 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb reported cstate change: term changed from 1 to 2, leader changed from 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) to ee8dcd502996434ea55a16a62a5324fb (127.16.122.5). New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:28.618479 23033 ts_tablet_manager.cc:1888] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.171
I20240502 21:26:28.618748 23033 log.cc:1196] T d15373bff59348b9a65b42053e8df389 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/d15373bff59348b9a65b42053e8df389
I20240502 21:26:28.619858 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet d15373bff59348b9a65b42053e8df389 (table rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]) successfully deleted
W20240502 21:26:28.641013 21713 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44822: Illegal state: replica 16745bde14bc4a43b532e6bb828f0625 is not leader of this config: current role FOLLOWER
I20240502 21:26:28.686486 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12887227115743afacb541c753db51c9" candidate_uuid: "ee8dcd502996434ea55a16a62a5324fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: true dest_uuid: "16745bde14bc4a43b532e6bb828f0625"
I20240502 21:26:28.686764 21752 raft_consensus.cc:3037] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:28.690187 21752 raft_consensus.cc:2454] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8dcd502996434ea55a16a62a5324fb in term 2.
I20240502 21:26:28.695340 20696 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20240502 21:26:28.696616 22849 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240502 21:26:28.747370 21752 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20240502 21:26:28.748637 22983 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap replayed 1/1 log segments. Stats: ops{read=197 overwritten=0 applied=196 ignored=0} inserts{seen=877 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:28.757347 22983 tablet_bootstrap.cc:492] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: Bootstrap complete.
I20240502 21:26:28.760442 22983 ts_tablet_manager.cc:1367] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent bootstrapping tablet: real 0.825s user 0.291s sys 0.027s
I20240502 21:26:28.761375 22983 raft_consensus.cc:348] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:28.762368 22983 raft_consensus.cc:729] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b4edc0e5b5c04ac5a7fe3f99d86d314b, State: Initialized, Role: LEARNER
I20240502 21:26:28.762650 22983 consensus_queue.cc:260] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 196, Last appended: 2.197, Last appended by leader: 197, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: true } }
I20240502 21:26:28.768826 22983 ts_tablet_manager.cc:1398] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:26:28.772715 21904 tablet_copy_service.cc:342] P ee8dcd502996434ea55a16a62a5324fb: Request end of tablet copy session b4edc0e5b5c04ac5a7fe3f99d86d314b-2998322d830948ac858235f4fb7defca received from {username='slave'} at 127.16.122.7:59377
I20240502 21:26:28.772966 21904 tablet_copy_service.cc:433] P ee8dcd502996434ea55a16a62a5324fb: ending tablet copy session b4edc0e5b5c04ac5a7fe3f99d86d314b-2998322d830948ac858235f4fb7defca on tablet 2998322d830948ac858235f4fb7defca with peer b4edc0e5b5c04ac5a7fe3f99d86d314b
I20240502 21:26:28.781443 22901 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240502 21:26:28.821127 21883 consensus_queue.cc:237] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 202, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } }
I20240502 21:26:28.831168 21752 raft_consensus.cc:1261] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20240502 21:26:28.842280 23048 consensus_queue.cc:1035] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240502 21:26:28.854215 23048 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 195 to 208, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.862286 20287 catalog_manager.cc:5484] T 12887227115743afacb541c753db51c9 P ee8dcd502996434ea55a16a62a5324fb reported cstate change: config changed from index 195 to 208, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:28.863346 21752 raft_consensus.cc:2932] T 12887227115743afacb541c753db51c9 P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index 195 to 208, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } }
I20240502 21:26:28.867317 20274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12887227115743afacb541c753db51c9 with cas_config_opid_index 195: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:28.877485 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet 12887227115743afacb541c753db51c9 with delete_type TABLET_DATA_TOMBSTONED (TS 0ac8c8b961dc4500bc8b9f1055298244 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:50304
I20240502 21:26:28.921761 23059 tablet_replica.cc:330] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: stopping tablet replica
I20240502 21:26:28.923048 23059 raft_consensus.cc:2229] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:28.924126 23059 raft_consensus.cc:2258] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:28.927073 23059 ts_tablet_manager.cc:1875] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:28.944202 23059 ts_tablet_manager.cc:1888] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20240502 21:26:28.945220 23059 log.cc:1196] T 12887227115743afacb541c753db51c9 P 0ac8c8b961dc4500bc8b9f1055298244: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/12887227115743afacb541c753db51c9
I20240502 21:26:28.947360 20273 catalog_manager.cc:4830] TS 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): tablet 12887227115743afacb541c753db51c9 (table rebalance_test_table_3 [id=7341cc2226374b2cb9e851ee2d2b2ef2]) successfully deleted
I20240502 21:26:29.189028 22147 raft_consensus.cc:1203] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.178->[2.179-2.220] Dedup: 2.197->[2.198-2.220]
I20240502 21:26:29.271896 22901 raft_consensus.cc:1050] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb: attempting to promote NON_VOTER b4edc0e5b5c04ac5a7fe3f99d86d314b to VOTER
I20240502 21:26:29.272974 22901 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 2.221, Last appended by leader: 117, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:29.275962 22148 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 LEARNER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20240502 21:26:29.278231 20427 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20240502 21:26:29.278558 22901 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:29.279698 22899 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:29.285689 22849 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.222: config changed from index 179 to 222, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:29.286856 22147 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 179 to 222, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:29.294288 20278 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b reported cstate change: config changed from index 179 to 222, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:29.295629 20427 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 179 to 222, b4edc0e5b5c04ac5a7fe3f99d86d314b (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:29.303439 21884 consensus_queue.cc:237] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 117, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } }
I20240502 21:26:29.305404 22148 raft_consensus.cc:1261] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20240502 21:26:29.306792 22901 consensus_queue.cc:1035] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240502 21:26:29.310684 22901 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 222 to 223, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:29.314026 22148 raft_consensus.cc:2932] T 2998322d830948ac858235f4fb7defca P b4edc0e5b5c04ac5a7fe3f99d86d314b [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 222 to 223, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } } }
I20240502 21:26:29.317283 20274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2998322d830948ac858235f4fb7defca with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:29.325033 20278 catalog_manager.cc:5484] T 2998322d830948ac858235f4fb7defca P ee8dcd502996434ea55a16a62a5324fb reported cstate change: config changed from index 222 to 223, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4edc0e5b5c04ac5a7fe3f99d86d314b" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 34365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:29.337853 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2998322d830948ac858235f4fb7defca with delete_type TABLET_DATA_TOMBSTONED (TS 35812e2216cb4758aee1e94877a823b7 not found in new config with opid_index 223) from {username='slave'} at 127.0.0.1:34202
I20240502 21:26:29.349694 23079 tablet_replica.cc:330] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: stopping tablet replica
I20240502 21:26:29.351197 23079 raft_consensus.cc:2229] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:29.357550 23079 raft_consensus.cc:2258] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:29.362059 23079 ts_tablet_manager.cc:1875] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:29.379352 23079 ts_tablet_manager.cc:1888] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.222
I20240502 21:26:29.379551 23079 log.cc:1196] T 2998322d830948ac858235f4fb7defca P 35812e2216cb4758aee1e94877a823b7: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/2998322d830948ac858235f4fb7defca
I20240502 21:26:29.380666 20274 catalog_manager.cc:4830] TS 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): tablet 2998322d830948ac858235f4fb7defca (table rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]) successfully deleted
I20240502 21:26:30.220149 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:30.230666 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:30.254976 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:30.288884 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:30.309425 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:30.358013 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:30.410982 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:31.546294 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:31.558709 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:31.578321 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:31.586244 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:31.630759 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:31.650733 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:31.693470 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:26:31.926311 20635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.6:38855, user_credentials={real_user=slave}} blocked reactor thread for 91327us
I20240502 21:26:34.257643 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:34.270998 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:34.272877 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:34.301054 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:34.306967 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:34.316520 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:34.531033 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:35.182456 20632 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:46571, user_credentials={real_user=slave}} blocked reactor thread for 48448us
I20240502 21:26:35.640715 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:35.710922 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:35.715512 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:35.722574 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:35.727830 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:35.751791 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:35.833503 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:36.368429 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:36.372390 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:36.376000 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:36.392812 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:36.409175 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:36.427840 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:36.470221 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ff826cbe705f48c5b303117140a0e6d2 | 127.16.122.62:40689 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.122.20:39533 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0ac8c8b961dc4500bc8b9f1055298244 | 127.16.122.3:34769 | HEALTHY | <none> | 4 | 0
16745bde14bc4a43b532e6bb828f0625 | 127.16.122.4:46571 | HEALTHY | <none> | 1 | 0
35812e2216cb4758aee1e94877a823b7 | 127.16.122.1:39539 | HEALTHY | <none> | 3 | 0
654f8d3e16bb4a658607a5c26f261cbf | 127.16.122.2:40149 | HEALTHY | <none> | 3 | 0
6c0aa2a3f325429eb83423bff3a80761 | 127.16.122.6:38855 | HEALTHY | <none> | 0 | 0
b4edc0e5b5c04ac5a7fe3f99d86d314b | 127.16.122.7:34365 | HEALTHY | <none> | 2 | 0
ee8dcd502996434ea55a16a62a5324fb | 127.16.122.5:35311 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.16.122.1 | experimental | 127.16.122.1:39539
local_ip_for_outbound_sockets | 127.16.122.2 | experimental | 127.16.122.2:40149
local_ip_for_outbound_sockets | 127.16.122.3 | experimental | 127.16.122.3:34769
local_ip_for_outbound_sockets | 127.16.122.4 | experimental | 127.16.122.4:46571
local_ip_for_outbound_sockets | 127.16.122.5 | experimental | 127.16.122.5:35311
local_ip_for_outbound_sockets | 127.16.122.6 | experimental | 127.16.122.6:38855
local_ip_for_outbound_sockets | 127.16.122.7 | experimental | 127.16.122.7:34365
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.122.1:39539
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.122.2:40149
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.122.3:34769
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.122.4:46571
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.122.5:35311
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.122.6:38855
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.122.7:34365
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.122.20:39533 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:26:36.532506 16872 log_verifier.cc:126] Checking tablet 028bf9972bf44bbbad3b862c70b174ca
I20240502 21:26:36.648080 16872 log_verifier.cc:177] Verified matching terms for 288 ops in tablet 028bf9972bf44bbbad3b862c70b174ca
I20240502 21:26:36.648545 16872 log_verifier.cc:126] Checking tablet 0e14655d22984ef3aab11be93eaaa07d
I20240502 21:26:36.665354 16872 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 0e14655d22984ef3aab11be93eaaa07d
I20240502 21:26:36.665475 16872 log_verifier.cc:126] Checking tablet 12887227115743afacb541c753db51c9
I20240502 21:26:36.799134 16872 log_verifier.cc:177] Verified matching terms for 329 ops in tablet 12887227115743afacb541c753db51c9
I20240502 21:26:36.799602 16872 log_verifier.cc:126] Checking tablet 129cc6d483b842fead09a5b1ddd58f52
I20240502 21:26:36.922166 16872 log_verifier.cc:177] Verified matching terms for 282 ops in tablet 129cc6d483b842fead09a5b1ddd58f52
I20240502 21:26:36.922617 16872 log_verifier.cc:126] Checking tablet 2998322d830948ac858235f4fb7defca
I20240502 21:26:37.085498 16872 log_verifier.cc:177] Verified matching terms for 329 ops in tablet 2998322d830948ac858235f4fb7defca
I20240502 21:26:37.085990 16872 log_verifier.cc:126] Checking tablet 380fd60c78334d90b26c3c9ce4945247
I20240502 21:26:37.238682 16872 log_verifier.cc:177] Verified matching terms for 289 ops in tablet 380fd60c78334d90b26c3c9ce4945247
I20240502 21:26:37.239122 16872 log_verifier.cc:126] Checking tablet 3fabe357cd7d42259e10511baa31b0bc
I20240502 21:26:37.383744 16872 log_verifier.cc:177] Verified matching terms for 281 ops in tablet 3fabe357cd7d42259e10511baa31b0bc
I20240502 21:26:37.384146 16872 log_verifier.cc:126] Checking tablet 44116b77039245b4883892cf19a8a248
I20240502 21:26:37.517212 16872 log_verifier.cc:177] Verified matching terms for 288 ops in tablet 44116b77039245b4883892cf19a8a248
I20240502 21:26:37.517648 16872 log_verifier.cc:126] Checking tablet 7875028620db4a8bbad7f0c5d1331f5e
I20240502 21:26:37.664000 16872 log_verifier.cc:177] Verified matching terms for 343 ops in tablet 7875028620db4a8bbad7f0c5d1331f5e
I20240502 21:26:37.664484 16872 log_verifier.cc:126] Checking tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f
I20240502 21:26:37.786530 16872 log_verifier.cc:177] Verified matching terms for 276 ops in tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f
I20240502 21:26:37.786921 16872 log_verifier.cc:126] Checking tablet 99a0d6a8a5a348dba1e8fd42d46b5f61
I20240502 21:26:37.941247 16872 log_verifier.cc:177] Verified matching terms for 257 ops in tablet 99a0d6a8a5a348dba1e8fd42d46b5f61
I20240502 21:26:37.941658 16872 log_verifier.cc:126] Checking tablet 9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:38.084425 16872 log_verifier.cc:177] Verified matching terms for 310 ops in tablet 9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:38.084868 16872 log_verifier.cc:126] Checking tablet d0892742cff44146ae2fb345d7e2bd3a
I20240502 21:26:38.228123 16872 log_verifier.cc:177] Verified matching terms for 307 ops in tablet d0892742cff44146ae2fb345d7e2bd3a
I20240502 21:26:38.228550 16872 log_verifier.cc:126] Checking tablet d15373bff59348b9a65b42053e8df389
I20240502 21:26:38.369251 16872 log_verifier.cc:177] Verified matching terms for 296 ops in tablet d15373bff59348b9a65b42053e8df389
I20240502 21:26:38.369746 16872 log_verifier.cc:126] Checking tablet d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:38.510242 16872 log_verifier.cc:177] Verified matching terms for 293 ops in tablet d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:38.510720 16872 log_verifier.cc:126] Checking tablet ef774f42c72349d0aa9d490bb5dd8011
I20240502 21:26:38.645051 16872 log_verifier.cc:177] Verified matching terms for 303 ops in tablet ef774f42c72349d0aa9d490bb5dd8011
I20240502 21:26:38.648985 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.8:0
--local_ip_for_outbound_sockets=127.16.122.8
--webserver_interface=127.16.122.8
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--builtin_ntp_servers=127.16.122.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:26:39.075685 23409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:39.076062 23409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:39.076213 23409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:39.086290 23409 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:39.086539 23409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:39.086796 23409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.8
I20240502 21:26:39.097326 23409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.122.8
--webserver_port=0
--tserver_master_addrs=127.16.122.62:40689
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.8
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:39.098789 23409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:39.100701 23409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:39.115374 23415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:39.115545 23418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:39.116020 23416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:39.116403 23409 server_base.cc:1034] running on GCE node
I20240502 21:26:39.117121 23409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:39.118296 23409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:39.119591 23409 hybrid_clock.cc:648] HybridClock initialized: now 1714685199119520 us; error 63 us; skew 500 ppm
I20240502 21:26:39.119954 23409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:39.128209 23409 webserver.cc:460] Webserver started at http://127.16.122.8:33367/ using document root <none> and password file <none>
I20240502 21:26:39.129081 23409 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:39.129215 23409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:39.129623 23409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:39.132441 23409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0230de442a8d4b9a968eb35462229e33"
format_stamp: "Formatted at 2024-05-02 21:26:39 on dist-test-slave-p9xn"
I20240502 21:26:39.133247 23409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0230de442a8d4b9a968eb35462229e33"
format_stamp: "Formatted at 2024-05-02 21:26:39 on dist-test-slave-p9xn"
I20240502 21:26:39.139544 23409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20240502 21:26:39.143698 23424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:39.145469 23409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:39.145651 23409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0230de442a8d4b9a968eb35462229e33"
format_stamp: "Formatted at 2024-05-02 21:26:39 on dist-test-slave-p9xn"
I20240502 21:26:39.145877 23409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:39.170503 23409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:39.171489 23409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:39.171784 23409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:39.172962 23409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:39.174582 23409 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:39.174685 23409 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:39.174808 23409 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:39.174888 23409 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:39.225124 23409 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.8:40417
I20240502 21:26:39.225194 23538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.8:40417 every 8 connection(s)
I20240502 21:26:39.226778 23409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:39.235878 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 23409
I20240502 21:26:39.236076 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:26:39.248730 23539 heartbeater.cc:347] Connected to a master server at 127.16.122.62:40689
I20240502 21:26:39.249238 23539 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:39.250572 23539 heartbeater.cc:511] Master 127.16.122.62:40689 requested a full tablet report, sending...
I20240502 21:26:39.252347 20280 ts_manager.cc:194] Registered new tserver with Master: 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417)
I20240502 21:26:39.253496 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.8:39087
I20240502 21:26:39.867774 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:39.872848 23474 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:39.883847 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:39.899310 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:39.956456 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:39.959152 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:39.959326 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:39.961285 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:40.200251 23474 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:40.218077 20678 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:26:40.228932 21733 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:40.239676 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:40.256970 23539 heartbeater.cc:503] Master 127.16.122.62:40689 was elected leader, sending a full tablet report...
I20240502 21:26:40.269346 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:40.276692 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:40.282835 21864 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:40.282868 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:40.386314 20426 consensus_queue.cc:237] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:40.399911 20564 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch)
I20240502 21:26:40.400991 23637 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
W20240502 21:26:40.410854 20362 consensus_peers.cc:485] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 -> Peer 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417): Couldn't send request to peer 0230de442a8d4b9a968eb35462229e33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7875028620db4a8bbad7f0c5d1331f5e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:40.412917 23312 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index -1 to 344, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.413723 20564 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index -1 to 344, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.419493 20284 catalog_manager.cc:5484] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index -1 to 344, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.435317 20698 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 2.310, Last appended by leader: 122, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:40.438463 21752 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 2 index: 310. Preceding OpId from leader: term: 2 index: 311. (index mismatch)
I20240502 21:26:40.439229 23311 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20240502 21:26:40.442638 23588 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Committing config change with OpId 2.311: config changed from index 125 to 311, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.443356 21752 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Committing config change with OpId 2.311: config changed from index 125 to 311, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
W20240502 21:26:40.445750 20631 consensus_peers.cc:485] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 -> Peer 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417): Couldn't send request to peer 0230de442a8d4b9a968eb35462229e33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ca7cd6b72d4419aade387eaf375454e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:40.448989 20281 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: config changed from index 125 to 311, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New cstate: current_term: 2 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.462559 21883 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 2.293, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:40.465590 20698 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20240502 21:26:40.466236 23589 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240502 21:26:40.469780 23589 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.294: config changed from index 110 to 294, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.474298 20698 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Committing config change with OpId 2.294: config changed from index 110 to 294, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
W20240502 21:26:40.474802 21817 consensus_peers.cc:485] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb -> Peer 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417): Couldn't send request to peer 0230de442a8d4b9a968eb35462229e33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4aeebd7828643ccab4aa62d4fd4d12a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:40.476256 20281 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb reported cstate change: config changed from index 110 to 294, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:40.501821 20564 consensus_queue.cc:237] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:40.504935 20426 raft_consensus.cc:1261] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 21:26:40.505677 23248 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 21:26:40.510001 20426 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index -1 to 258, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.509128 23668 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index -1 to 258, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
W20240502 21:26:40.512207 20494 consensus_peers.cc:485] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf -> Peer 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417): Couldn't send request to peer 0230de442a8d4b9a968eb35462229e33. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a0d6a8a5a348dba1e8fd42d46b5f61. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:40.515975 20284 catalog_manager.cc:5484] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 reported cstate change: config changed from index -1 to 258, NON_VOTER 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } } }
I20240502 21:26:40.892830 23711 ts_tablet_manager.cc:902] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Initiating tablet copy from peer ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311)
I20240502 21:26:40.895442 23711 tablet_copy_client.cc:308] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.5:35311
I20240502 21:26:40.904016 21904 tablet_copy_service.cc:140] P ee8dcd502996434ea55a16a62a5324fb: Received BeginTabletCopySession request for tablet d4aeebd7828643ccab4aa62d4fd4d12a from peer 0230de442a8d4b9a968eb35462229e33 ({username='slave'} at 127.16.122.8:39187)
I20240502 21:26:40.904333 21904 tablet_copy_service.cc:161] P ee8dcd502996434ea55a16a62a5324fb: Beginning new tablet copy session on tablet d4aeebd7828643ccab4aa62d4fd4d12a from peer 0230de442a8d4b9a968eb35462229e33 at {username='slave'} at 127.16.122.8:39187: session id = 0230de442a8d4b9a968eb35462229e33-d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:40.907794 21904 tablet_copy_source_session.cc:205] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:40.913448 23711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aeebd7828643ccab4aa62d4fd4d12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.925469 23711 tablet_copy_client.cc:783] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:40.925843 23711 tablet_copy_client.cc:647] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:40.930840 23711 tablet_copy_client.cc:523] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:40.934880 23711 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Bootstrap starting.
I20240502 21:26:40.940412 23717 ts_tablet_manager.cc:902] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: Initiating tablet copy from peer 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149)
I20240502 21:26:40.942803 23717 tablet_copy_client.cc:308] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:40149
I20240502 21:26:40.949563 20585 tablet_copy_service.cc:140] P 654f8d3e16bb4a658607a5c26f261cbf: Received BeginTabletCopySession request for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 from peer 0230de442a8d4b9a968eb35462229e33 ({username='slave'} at 127.16.122.8:50319)
I20240502 21:26:40.949793 20585 tablet_copy_service.cc:161] P 654f8d3e16bb4a658607a5c26f261cbf: Beginning new tablet copy session on tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 from peer 0230de442a8d4b9a968eb35462229e33 at {username='slave'} at 127.16.122.8:50319: session id = 0230de442a8d4b9a968eb35462229e33-99a0d6a8a5a348dba1e8fd42d46b5f61
I20240502 21:26:40.953130 20585 tablet_copy_source_session.cc:205] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:40.954860 23717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:40.960215 23717 tablet_copy_client.cc:783] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:40.960510 23717 tablet_copy_client.cc:647] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:40.963416 23717 tablet_copy_client.cc:523] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:40.966619 23717 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: Bootstrap starting.
I20240502 21:26:40.995311 23722 ts_tablet_manager.cc:902] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: Initiating tablet copy from peer 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
I20240502 21:26:41.002054 23711 log.cc:824] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:41.010412 23726 ts_tablet_manager.cc:902] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: Initiating tablet copy from peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769)
I20240502 21:26:41.027179 23722 tablet_copy_client.cc:308] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:39539
I20240502 21:26:41.038465 20449 tablet_copy_service.cc:140] P 35812e2216cb4758aee1e94877a823b7: Received BeginTabletCopySession request for tablet 7875028620db4a8bbad7f0c5d1331f5e from peer 0230de442a8d4b9a968eb35462229e33 ({username='slave'} at 127.16.122.8:45791)
I20240502 21:26:41.038897 20449 tablet_copy_service.cc:161] P 35812e2216cb4758aee1e94877a823b7: Beginning new tablet copy session on tablet 7875028620db4a8bbad7f0c5d1331f5e from peer 0230de442a8d4b9a968eb35462229e33 at {username='slave'} at 127.16.122.8:45791: session id = 0230de442a8d4b9a968eb35462229e33-7875028620db4a8bbad7f0c5d1331f5e
I20240502 21:26:41.040691 23726 tablet_copy_client.cc:308] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:34769
I20240502 21:26:41.050285 20719 tablet_copy_service.cc:140] P 0ac8c8b961dc4500bc8b9f1055298244: Received BeginTabletCopySession request for tablet 9ca7cd6b72d4419aade387eaf375454e from peer 0230de442a8d4b9a968eb35462229e33 ({username='slave'} at 127.16.122.8:34579)
I20240502 21:26:41.050593 20719 tablet_copy_service.cc:161] P 0ac8c8b961dc4500bc8b9f1055298244: Beginning new tablet copy session on tablet 9ca7cd6b72d4419aade387eaf375454e from peer 0230de442a8d4b9a968eb35462229e33 at {username='slave'} at 127.16.122.8:34579: session id = 0230de442a8d4b9a968eb35462229e33-9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:41.054281 20719 tablet_copy_source_session.cc:205] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:41.056006 20449 tablet_copy_source_session.cc:205] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:41.056752 23726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca7cd6b72d4419aade387eaf375454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:41.059610 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7875028620db4a8bbad7f0c5d1331f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:41.065613 23726 tablet_copy_client.cc:783] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:41.065958 23726 tablet_copy_client.cc:647] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:41.068017 23722 tablet_copy_client.cc:783] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:41.068459 23722 tablet_copy_client.cc:647] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:41.075400 23726 tablet_copy_client.cc:523] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:41.084502 23726 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: Bootstrap starting.
I20240502 21:26:41.089670 23722 tablet_copy_client.cc:523] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:41.094281 23722 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: Bootstrap starting.
I20240502 21:26:41.555420 23717 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: Bootstrap replayed 1/1 log segments. Stats: ops{read=258 overwritten=0 applied=258 ignored=0} inserts{seen=930 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:41.556238 23711 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Bootstrap replayed 1/1 log segments. Stats: ops{read=294 overwritten=0 applied=294 ignored=0} inserts{seen=975 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:41.556982 23711 tablet_bootstrap.cc:492] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Bootstrap complete.
I20240502 21:26:41.557621 23717 tablet_bootstrap.cc:492] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: Bootstrap complete.
I20240502 21:26:41.557760 23711 ts_tablet_manager.cc:1367] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Time spent bootstrapping tablet: real 0.623s user 0.391s sys 0.051s
I20240502 21:26:41.558406 23717 ts_tablet_manager.cc:1367] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: Time spent bootstrapping tablet: real 0.592s user 0.365s sys 0.023s
I20240502 21:26:41.568578 23717 raft_consensus.cc:348] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.569396 23717 raft_consensus.cc:729] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0230de442a8d4b9a968eb35462229e33, State: Initialized, Role: LEARNER
I20240502 21:26:41.570113 23717 consensus_queue.cc:260] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 258, Last appended: 1.258, Last appended by leader: 258, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.573551 23711 raft_consensus.cc:348] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.575480 23711 raft_consensus.cc:729] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0230de442a8d4b9a968eb35462229e33, State: Initialized, Role: LEARNER
I20240502 21:26:41.576623 23711 consensus_queue.cc:260] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 294, Last appended: 2.294, Last appended by leader: 294, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.584293 23711 ts_tablet_manager.cc:1398] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33: Time spent starting tablet: real 0.026s user 0.002s sys 0.002s
I20240502 21:26:41.585578 21904 tablet_copy_service.cc:342] P ee8dcd502996434ea55a16a62a5324fb: Request end of tablet copy session 0230de442a8d4b9a968eb35462229e33-d4aeebd7828643ccab4aa62d4fd4d12a received from {username='slave'} at 127.16.122.8:39187
I20240502 21:26:41.585737 21904 tablet_copy_service.cc:433] P ee8dcd502996434ea55a16a62a5324fb: ending tablet copy session 0230de442a8d4b9a968eb35462229e33-d4aeebd7828643ccab4aa62d4fd4d12a on tablet d4aeebd7828643ccab4aa62d4fd4d12a with peer 0230de442a8d4b9a968eb35462229e33
I20240502 21:26:41.592818 23717 ts_tablet_manager.cc:1398] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33: Time spent starting tablet: real 0.034s user 0.013s sys 0.003s
I20240502 21:26:41.593748 20585 tablet_copy_service.cc:342] P 654f8d3e16bb4a658607a5c26f261cbf: Request end of tablet copy session 0230de442a8d4b9a968eb35462229e33-99a0d6a8a5a348dba1e8fd42d46b5f61 received from {username='slave'} at 127.16.122.8:50319
I20240502 21:26:41.593886 20585 tablet_copy_service.cc:433] P 654f8d3e16bb4a658607a5c26f261cbf: ending tablet copy session 0230de442a8d4b9a968eb35462229e33-99a0d6a8a5a348dba1e8fd42d46b5f61 on tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 with peer 0230de442a8d4b9a968eb35462229e33
I20240502 21:26:41.657310 23726 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: Bootstrap replayed 1/1 log segments. Stats: ops{read=311 overwritten=0 applied=311 ignored=0} inserts{seen=938 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:41.658044 23726 tablet_bootstrap.cc:492] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: Bootstrap complete.
I20240502 21:26:41.658434 23726 ts_tablet_manager.cc:1367] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: Time spent bootstrapping tablet: real 0.574s user 0.413s sys 0.040s
I20240502 21:26:41.659363 23726 raft_consensus.cc:348] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.659646 23726 raft_consensus.cc:729] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0230de442a8d4b9a968eb35462229e33, State: Initialized, Role: LEARNER
I20240502 21:26:41.659960 23726 consensus_queue.cc:260] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 311, Last appended: 2.311, Last appended by leader: 311, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.660780 23726 ts_tablet_manager.cc:1398] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:41.662088 20719 tablet_copy_service.cc:342] P 0ac8c8b961dc4500bc8b9f1055298244: Request end of tablet copy session 0230de442a8d4b9a968eb35462229e33-9ca7cd6b72d4419aade387eaf375454e received from {username='slave'} at 127.16.122.8:34579
I20240502 21:26:41.662262 20719 tablet_copy_service.cc:433] P 0ac8c8b961dc4500bc8b9f1055298244: ending tablet copy session 0230de442a8d4b9a968eb35462229e33-9ca7cd6b72d4419aade387eaf375454e on tablet 9ca7cd6b72d4419aade387eaf375454e with peer 0230de442a8d4b9a968eb35462229e33
I20240502 21:26:41.668278 23722 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: Bootstrap replayed 1/1 log segments. Stats: ops{read=344 overwritten=0 applied=344 ignored=0} inserts{seen=1016 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:41.668699 23722 tablet_bootstrap.cc:492] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: Bootstrap complete.
I20240502 21:26:41.668991 23722 ts_tablet_manager.cc:1367] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: Time spent bootstrapping tablet: real 0.575s user 0.431s sys 0.054s
I20240502 21:26:41.669636 23722 raft_consensus.cc:348] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.669833 23722 raft_consensus.cc:729] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0230de442a8d4b9a968eb35462229e33, State: Initialized, Role: LEARNER
I20240502 21:26:41.669976 23722 consensus_queue.cc:260] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 344, Last appended: 1.344, Last appended by leader: 344, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: true } }
I20240502 21:26:41.670440 23722 ts_tablet_manager.cc:1398] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 21:26:41.671490 20449 tablet_copy_service.cc:342] P 35812e2216cb4758aee1e94877a823b7: Request end of tablet copy session 0230de442a8d4b9a968eb35462229e33-7875028620db4a8bbad7f0c5d1331f5e received from {username='slave'} at 127.16.122.8:45791
I20240502 21:26:41.671722 20449 tablet_copy_service.cc:433] P 35812e2216cb4758aee1e94877a823b7: ending tablet copy session 0230de442a8d4b9a968eb35462229e33-7875028620db4a8bbad7f0c5d1331f5e on tablet 7875028620db4a8bbad7f0c5d1331f5e with peer 0230de442a8d4b9a968eb35462229e33
I20240502 21:26:41.917439 23494 raft_consensus.cc:1203] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.293->[2.294-2.294] Dedup: 2.294->[]
I20240502 21:26:42.011394 23494 raft_consensus.cc:1203] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.257->[1.258-1.258] Dedup: 1.258->[]
I20240502 21:26:42.134625 23494 raft_consensus.cc:1203] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.310->[2.311-2.311] Dedup: 2.311->[]
I20240502 21:26:42.182739 23494 raft_consensus.cc:1203] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.343->[1.344-1.344] Dedup: 1.344->[]
I20240502 21:26:42.386942 23806 raft_consensus.cc:1050] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: attempting to promote NON_VOTER 0230de442a8d4b9a968eb35462229e33 to VOTER
I20240502 21:26:42.387655 23806 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.390167 20698 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20240502 21:26:42.390473 23494 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Refusing update from remote peer ee8dcd502996434ea55a16a62a5324fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20240502 21:26:42.390790 23757 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20240502 21:26:42.391227 23806 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20240502 21:26:42.394062 23806 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 294 to 295, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.397284 20698 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.402144 20281 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 294 to 295, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ee8dcd502996434ea55a16a62a5324fb" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.405148 23494 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.568670 23765 raft_consensus.cc:1050] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: attempting to promote NON_VOTER 0230de442a8d4b9a968eb35462229e33 to VOTER
I20240502 21:26:42.569586 23765 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 122, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.573813 23494 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 2 LEARNER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20240502 21:26:42.573886 21752 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Refusing update from remote peer 0ac8c8b961dc4500bc8b9f1055298244: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20240502 21:26:42.574594 23829 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240502 21:26:42.575135 23830 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240502 21:26:42.579473 23829 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 311 to 312, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.580819 23494 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.583042 21752 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.586004 20278 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 reported cstate change: config changed from index 311 to 312, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0ac8c8b961dc4500bc8b9f1055298244" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:42.603111 23873 raft_consensus.cc:1050] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: attempting to promote NON_VOTER 0230de442a8d4b9a968eb35462229e33 to VOTER
I20240502 21:26:42.603986 23873 consensus_queue.cc:237] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 1.344, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.606928 23494 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20240502 21:26:42.607160 20564 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20240502 21:26:42.607522 23777 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20240502 21:26:42.608531 23777 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20240502 21:26:42.612265 23831 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Committing config change with OpId 1.345: config changed from index 344 to 345, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.613485 20564 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.618393 20284 catalog_manager.cc:5484] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 344 to 345, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.618643 23494 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.632820 23882 raft_consensus.cc:1050] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: attempting to promote NON_VOTER 0230de442a8d4b9a968eb35462229e33 to VOTER
I20240502 21:26:42.633913 23882 consensus_queue.cc:237] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.636683 23494 raft_consensus.cc:1261] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 1 LEARNER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240502 21:26:42.636669 20426 raft_consensus.cc:1261] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240502 21:26:42.637248 23885 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240502 21:26:42.637650 23881 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240502 21:26:42.640486 23881 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 258 to 259, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.641654 21883 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a"
dest_uuid: "ee8dcd502996434ea55a16a62a5324fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50752
I20240502 21:26:42.641920 21883 raft_consensus.cc:595] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Received request to transfer leadership
I20240502 21:26:42.642648 23494 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.642788 20426 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:42.647508 20278 catalog_manager.cc:5484] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 258 to 259, 0230de442a8d4b9a968eb35462229e33 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:42.667172 20426 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e"
dest_uuid: "35812e2216cb4758aee1e94877a823b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42122
I20240502 21:26:42.667449 20426 raft_consensus.cc:595] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:42.687844 20698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e"
dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48494
I20240502 21:26:42.688056 20698 raft_consensus.cc:595] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:26:42.822026 23757 raft_consensus.cc:979] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: : Instructing follower 0230de442a8d4b9a968eb35462229e33 to start an election
I20240502 21:26:42.822228 23757 raft_consensus.cc:1067] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Signalling peer 0230de442a8d4b9a968eb35462229e33 to start an election
I20240502 21:26:42.823102 23494 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a"
dest_uuid: "0230de442a8d4b9a968eb35462229e33"
from {username='slave'} at 127.16.122.5:46511
I20240502 21:26:42.823423 23494 raft_consensus.cc:482] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:42.823585 23494 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:26:42.825742 23494 raft_consensus.cc:504] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.827373 23494 leader_election.cc:288] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [CANDIDATE]: Term 3 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769), ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311)
I20240502 21:26:42.827853 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a" candidate_uuid: "0230de442a8d4b9a968eb35462229e33" candidate_term: 3 candidate_status { last_received { term: 2 index: 295 } } ignore_live_leader: true dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:42.828116 20698 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:26:42.828032 21883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aeebd7828643ccab4aa62d4fd4d12a" candidate_uuid: "0230de442a8d4b9a968eb35462229e33" candidate_term: 3 candidate_status { last_received { term: 2 index: 295 } } ignore_live_leader: true dest_uuid: "ee8dcd502996434ea55a16a62a5324fb"
I20240502 21:26:42.828316 21883 raft_consensus.cc:3032] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:26:42.828438 21883 raft_consensus.cc:729] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee8dcd502996434ea55a16a62a5324fb, State: Running, Role: LEADER
I20240502 21:26:42.828766 21883 consensus_queue.cc:260] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 295, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.829274 21883 raft_consensus.cc:3037] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:26:42.831169 20698 raft_consensus.cc:2454] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0230de442a8d4b9a968eb35462229e33 in term 3.
I20240502 21:26:42.831951 23427 leader_election.cc:302] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0230de442a8d4b9a968eb35462229e33, 0ac8c8b961dc4500bc8b9f1055298244; no voters:
I20240502 21:26:42.832429 23771 raft_consensus.cc:2781] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:26:42.832453 21883 raft_consensus.cc:2454] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0230de442a8d4b9a968eb35462229e33 in term 3.
I20240502 21:26:42.833161 23771 raft_consensus.cc:686] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 3 LEADER]: Becoming Leader. State: Replica: 0230de442a8d4b9a968eb35462229e33, State: Running, Role: LEADER
I20240502 21:26:42.833715 23771 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 295, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:42.837235 20281 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 reported cstate change: term changed from 2 to 3, leader changed from ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) to 0230de442a8d4b9a968eb35462229e33 (127.16.122.8). New cstate: current_term: 3 leader_uuid: "0230de442a8d4b9a968eb35462229e33" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:42.916077 20564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a0d6a8a5a348dba1e8fd42d46b5f61"
dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54600
I20240502 21:26:42.916422 20564 raft_consensus.cc:595] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:42.947255 20426 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e"
dest_uuid: "35812e2216cb4758aee1e94877a823b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42140
I20240502 21:26:42.947484 20426 raft_consensus.cc:595] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:42.966811 20698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e"
dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48504
I20240502 21:26:42.967023 20698 raft_consensus.cc:595] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:26:42.997267 23829 raft_consensus.cc:979] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: : Instructing follower 16745bde14bc4a43b532e6bb828f0625 to start an election
I20240502 21:26:42.997515 23829 raft_consensus.cc:1067] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Signalling peer 16745bde14bc4a43b532e6bb828f0625 to start an election
I20240502 21:26:42.998217 21752 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e"
dest_uuid: "16745bde14bc4a43b532e6bb828f0625"
from {username='slave'} at 127.16.122.3:56151
I20240502 21:26:42.998463 21752 raft_consensus.cc:482] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:42.998607 21752 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:26:43.000598 21752 raft_consensus.cc:504] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.001659 21752 leader_election.cc:288] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [CANDIDATE]: Term 3 election: Requested vote from peers 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769), 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417)
I20240502 21:26:43.002202 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e" candidate_uuid: "16745bde14bc4a43b532e6bb828f0625" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: true dest_uuid: "0ac8c8b961dc4500bc8b9f1055298244"
I20240502 21:26:43.002537 20698 raft_consensus.cc:3032] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:26:43.002631 20698 raft_consensus.cc:729] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ac8c8b961dc4500bc8b9f1055298244, State: Running, Role: LEADER
I20240502 21:26:43.002941 20698 consensus_queue.cc:260] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 312, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.003300 20698 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:26:43.005268 20698 raft_consensus.cc:2454] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16745bde14bc4a43b532e6bb828f0625 in term 3.
I20240502 21:26:43.005712 21688 leader_election.cc:302] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac8c8b961dc4500bc8b9f1055298244, 16745bde14bc4a43b532e6bb828f0625; no voters:
I20240502 21:26:43.006050 23832 raft_consensus.cc:2781] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:26:43.006589 23832 raft_consensus.cc:686] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 3 LEADER]: Becoming Leader. State: Replica: 16745bde14bc4a43b532e6bb828f0625, State: Running, Role: LEADER
I20240502 21:26:43.006947 23832 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 312, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.008384 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca7cd6b72d4419aade387eaf375454e" candidate_uuid: "16745bde14bc4a43b532e6bb828f0625" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: true dest_uuid: "0230de442a8d4b9a968eb35462229e33"
I20240502 21:26:43.008674 23494 raft_consensus.cc:3037] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:26:43.009899 20281 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: term changed from 2 to 3, leader changed from 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) to 16745bde14bc4a43b532e6bb828f0625 (127.16.122.4). New cstate: current_term: 3 leader_uuid: "16745bde14bc4a43b532e6bb828f0625" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:43.011627 23494 raft_consensus.cc:2454] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16745bde14bc4a43b532e6bb828f0625 in term 3.
I20240502 21:26:43.028039 23831 raft_consensus.cc:979] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: : Instructing follower 654f8d3e16bb4a658607a5c26f261cbf to start an election
I20240502 21:26:43.028225 23831 raft_consensus.cc:1067] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Signalling peer 654f8d3e16bb4a658607a5c26f261cbf to start an election
I20240502 21:26:43.029112 20564 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e"
dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
from {username='slave'} at 127.16.122.1:57667
I20240502 21:26:43.029397 20564 raft_consensus.cc:482] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:43.029510 20564 raft_consensus.cc:3037] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:43.031847 20564 raft_consensus.cc:504] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.032720 20564 leader_election.cc:288] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 2 election: Requested vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539), 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417)
I20240502 21:26:43.033457 20426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 345 } } ignore_live_leader: true dest_uuid: "35812e2216cb4758aee1e94877a823b7"
I20240502 21:26:43.033437 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7875028620db4a8bbad7f0c5d1331f5e" candidate_uuid: "654f8d3e16bb4a658607a5c26f261cbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 345 } } ignore_live_leader: true dest_uuid: "0230de442a8d4b9a968eb35462229e33"
I20240502 21:26:43.033689 20426 raft_consensus.cc:3032] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:43.033675 23494 raft_consensus.cc:3037] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:43.033788 20426 raft_consensus.cc:729] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Running, Role: LEADER
I20240502 21:26:43.034140 20426 consensus_queue.cc:260] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 345, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.034668 20426 raft_consensus.cc:3037] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:43.036713 23494 raft_consensus.cc:2454] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 2.
I20240502 21:26:43.037048 23493 raft_consensus.cc:1226] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Rejecting Update request from peer 35812e2216cb4758aee1e94877a823b7 for earlier term 1. Current term is 2. Ops: []
I20240502 21:26:43.037251 20494 leader_election.cc:302] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0230de442a8d4b9a968eb35462229e33, 654f8d3e16bb4a658607a5c26f261cbf; no voters:
I20240502 21:26:43.037572 20426 raft_consensus.cc:2454] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654f8d3e16bb4a658607a5c26f261cbf in term 2.
I20240502 21:26:43.037667 23885 raft_consensus.cc:2781] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:43.037986 23885 raft_consensus.cc:686] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 LEADER]: Becoming Leader. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:43.038450 23885 consensus_queue.cc:237] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 345, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.041283 20281 catalog_manager.cc:5484] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: term changed from 1 to 2, leader changed from 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) to 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2). New cstate: current_term: 2 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:43.085888 23881 raft_consensus.cc:979] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: : Instructing follower 35812e2216cb4758aee1e94877a823b7 to start an election
I20240502 21:26:43.086117 23885 raft_consensus.cc:1067] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Signalling peer 35812e2216cb4758aee1e94877a823b7 to start an election
I20240502 21:26:43.086900 20426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a0d6a8a5a348dba1e8fd42d46b5f61"
dest_uuid: "35812e2216cb4758aee1e94877a823b7"
from {username='slave'} at 127.16.122.2:36075
I20240502 21:26:43.087114 20426 raft_consensus.cc:482] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:43.087209 20426 raft_consensus.cc:3037] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:43.089416 20426 raft_consensus.cc:504] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.090190 20426 leader_election.cc:288] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 election: Requested vote from peers 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149), 0230de442a8d4b9a968eb35462229e33 (127.16.122.8:40417)
I20240502 21:26:43.090934 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a0d6a8a5a348dba1e8fd42d46b5f61" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 259 } } ignore_live_leader: true dest_uuid: "654f8d3e16bb4a658607a5c26f261cbf"
I20240502 21:26:43.090866 23493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a0d6a8a5a348dba1e8fd42d46b5f61" candidate_uuid: "35812e2216cb4758aee1e94877a823b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 259 } } ignore_live_leader: true dest_uuid: "0230de442a8d4b9a968eb35462229e33"
I20240502 21:26:43.091205 20564 raft_consensus.cc:3032] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:43.091267 23493 raft_consensus.cc:3037] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:43.091332 20564 raft_consensus.cc:729] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 654f8d3e16bb4a658607a5c26f261cbf, State: Running, Role: LEADER
I20240502 21:26:43.091729 20564 consensus_queue.cc:260] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 1.259, Last appended by leader: 259, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.092227 20564 raft_consensus.cc:3037] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:43.094302 23493 raft_consensus.cc:2454] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 2.
I20240502 21:26:43.094902 20362 leader_election.cc:302] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0230de442a8d4b9a968eb35462229e33, 35812e2216cb4758aee1e94877a823b7; no voters:
I20240502 21:26:43.095266 23831 raft_consensus.cc:2781] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:43.095427 20564 raft_consensus.cc:2454] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35812e2216cb4758aee1e94877a823b7 in term 2.
I20240502 21:26:43.095461 23831 raft_consensus.cc:686] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 2 LEADER]: Becoming Leader. State: Replica: 35812e2216cb4758aee1e94877a823b7, State: Running, Role: LEADER
I20240502 21:26:43.095763 23831 consensus_queue.cc:237] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 1.259, Last appended by leader: 259, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.098441 20281 catalog_manager.cc:5484] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 reported cstate change: term changed from 1 to 2, leader changed from 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) to 35812e2216cb4758aee1e94877a823b7 (127.16.122.1). New cstate: current_term: 2 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:43.380291 20698 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Refusing update from remote peer 0230de442a8d4b9a968eb35462229e33: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 3 index: 296. (index mismatch)
I20240502 21:26:43.381089 23969 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240502 21:26:43.384893 20698 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Refusing update from remote peer 16745bde14bc4a43b532e6bb828f0625: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 3 index: 313. (index mismatch)
I20240502 21:26:43.386904 23832 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.001s
I20240502 21:26:43.390146 21883 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 3 FOLLOWER]: Refusing update from remote peer 0230de442a8d4b9a968eb35462229e33: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 3 index: 296. (index mismatch)
I20240502 21:26:43.391420 23970 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8dcd502996434ea55a16a62a5324fb" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 35311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240502 21:26:43.397734 23493 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 3 FOLLOWER]: Refusing update from remote peer 16745bde14bc4a43b532e6bb828f0625: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 3 index: 313. (index mismatch)
I20240502 21:26:43.399766 23832 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20240502 21:26:43.408052 23493 consensus_queue.cc:237] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 3.296, Last appended by leader: 295, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.409773 20696 raft_consensus.cc:1261] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Refusing update from remote peer 0230de442a8d4b9a968eb35462229e33: Log matching property violated. Preceding OpId in replica: term: 3 index: 296. Preceding OpId from leader: term: 3 index: 297. (index mismatch)
I20240502 21:26:43.410321 23970 consensus_queue.cc:1035] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240502 21:26:43.412575 23969 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 [term 3 LEADER]: Committing config change with OpId 3.297: config changed from index 295 to 297, VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:43.413249 20696 raft_consensus.cc:2932] T d4aeebd7828643ccab4aa62d4fd4d12a P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Committing config change with OpId 3.297: config changed from index 295 to 297, VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:43.417158 20271 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4aeebd7828643ccab4aa62d4fd4d12a with cas_config_opid_index 295: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:43.419039 20281 catalog_manager.cc:5484] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 reported cstate change: config changed from index 295 to 297, VOTER ee8dcd502996434ea55a16a62a5324fb (127.16.122.5) evicted. New cstate: current_term: 3 leader_uuid: "0230de442a8d4b9a968eb35462229e33" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "0ac8c8b961dc4500bc8b9f1055298244" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:43.423581 21864 tablet_service.cc:1514] Processing DeleteTablet for tablet d4aeebd7828643ccab4aa62d4fd4d12a with delete_type TABLET_DATA_TOMBSTONED (TS ee8dcd502996434ea55a16a62a5324fb not found in new config with opid_index 297) from {username='slave'} at 127.0.0.1:59364
I20240502 21:26:43.426621 23978 tablet_replica.cc:330] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: stopping tablet replica
I20240502 21:26:43.427218 23978 raft_consensus.cc:2229] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.427542 23978 raft_consensus.cc:2258] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.429637 23978 ts_tablet_manager.cc:1875] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:43.436002 23978 ts_tablet_manager.cc:1888] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.296
I20240502 21:26:43.436213 23978 log.cc:1196] T d4aeebd7828643ccab4aa62d4fd4d12a P ee8dcd502996434ea55a16a62a5324fb: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/wals/d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:43.437057 20274 catalog_manager.cc:4830] TS ee8dcd502996434ea55a16a62a5324fb (127.16.122.5:35311): tablet d4aeebd7828643ccab4aa62d4fd4d12a (table rebalance_test_table_4 [id=3ea13f305358431fbbafab8ba82d063f]) successfully deleted
I20240502 21:26:43.500849 20564 raft_consensus.cc:1261] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20240502 21:26:43.501417 23831 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20240502 21:26:43.506088 23493 raft_consensus.cc:1261] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20240502 21:26:43.506646 23873 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20240502 21:26:43.533922 20426 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20240502 21:26:43.534627 23885 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 21:26:43.538017 23493 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20240502 21:26:43.538980 23885 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 21:26:43.956641 21752 consensus_queue.cc:237] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 3.313, Last appended by leader: 312, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.958279 23493 raft_consensus.cc:1261] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 3 FOLLOWER]: Refusing update from remote peer 16745bde14bc4a43b532e6bb828f0625: Log matching property violated. Preceding OpId in replica: term: 3 index: 313. Preceding OpId from leader: term: 3 index: 314. (index mismatch)
I20240502 21:26:43.958822 24047 consensus_queue.cc:1035] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240502 21:26:43.960973 23832 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 [term 3 LEADER]: Committing config change with OpId 3.314: config changed from index 312 to 314, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:43.961513 23493 raft_consensus.cc:2932] T 9ca7cd6b72d4419aade387eaf375454e P 0230de442a8d4b9a968eb35462229e33 [term 3 FOLLOWER]: Committing config change with OpId 3.314: config changed from index 312 to 314, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:43.963896 20272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ca7cd6b72d4419aade387eaf375454e with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:43.965795 20281 catalog_manager.cc:5484] T 9ca7cd6b72d4419aade387eaf375454e P 16745bde14bc4a43b532e6bb828f0625 reported cstate change: config changed from index 312 to 314, VOTER 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3) evicted. New cstate: current_term: 3 leader_uuid: "16745bde14bc4a43b532e6bb828f0625" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "16745bde14bc4a43b532e6bb828f0625" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 46571 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:43.970077 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca7cd6b72d4419aade387eaf375454e with delete_type TABLET_DATA_TOMBSTONED (TS 0ac8c8b961dc4500bc8b9f1055298244 not found in new config with opid_index 314) from {username='slave'} at 127.0.0.1:50304
I20240502 21:26:43.970767 24051 tablet_replica.cc:330] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: stopping tablet replica
I20240502 21:26:43.970961 24051 raft_consensus.cc:2229] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.971230 24051 raft_consensus.cc:2258] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.972843 24051 ts_tablet_manager.cc:1875] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:43.977138 24051 ts_tablet_manager.cc:1888] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.313
I20240502 21:26:43.977258 24051 log.cc:1196] T 9ca7cd6b72d4419aade387eaf375454e P 0ac8c8b961dc4500bc8b9f1055298244: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:43.978036 20273 catalog_manager.cc:4830] TS 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): tablet 9ca7cd6b72d4419aade387eaf375454e (table rebalance_test_table_2 [id=b813afcdc0bb448499de47644d610fac]) successfully deleted
I20240502 21:26:43.991683 20426 consensus_queue.cc:237] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 2.260, Last appended by leader: 259, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:43.993366 23493 raft_consensus.cc:1261] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Refusing update from remote peer 35812e2216cb4758aee1e94877a823b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 260. Preceding OpId from leader: term: 2 index: 261. (index mismatch)
I20240502 21:26:43.994755 24013 consensus_queue.cc:1035] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20240502 21:26:43.997555 23873 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 35812e2216cb4758aee1e94877a823b7 [term 2 LEADER]: Committing config change with OpId 2.261: config changed from index 259 to 261, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:43.998661 23493 raft_consensus.cc:2932] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Committing config change with OpId 2.261: config changed from index 259 to 261, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:44.001397 20274 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 with cas_config_opid_index 259: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:44.003871 20281 catalog_manager.cc:5484] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 reported cstate change: config changed from index 259 to 261, VOTER 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "35812e2216cb4758aee1e94877a823b7" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:44.007297 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 with delete_type TABLET_DATA_TOMBSTONED (TS 654f8d3e16bb4a658607a5c26f261cbf not found in new config with opid_index 261) from {username='slave'} at 127.0.0.1:34210
I20240502 21:26:44.007941 24052 tablet_replica.cc:330] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: stopping tablet replica
I20240502 21:26:44.008137 24052 raft_consensus.cc:2229] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.008366 24052 raft_consensus.cc:2258] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.009948 24052 ts_tablet_manager.cc:1875] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:44.014894 24052 ts_tablet_manager.cc:1888] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.260
I20240502 21:26:44.015040 24052 log.cc:1196] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 654f8d3e16bb4a658607a5c26f261cbf: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/99a0d6a8a5a348dba1e8fd42d46b5f61
I20240502 21:26:44.015923 20272 catalog_manager.cc:4830] TS 654f8d3e16bb4a658607a5c26f261cbf (127.16.122.2:40149): tablet 99a0d6a8a5a348dba1e8fd42d46b5f61 (table rebalance_test_table_0 [id=3fe7293eed8344718cbd9635a4cc2fe7]) successfully deleted
I20240502 21:26:44.116035 20564 consensus_queue.cc:237] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 2.346, Last appended by leader: 345, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:44.117700 23493 raft_consensus.cc:1261] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Refusing update from remote peer 654f8d3e16bb4a658607a5c26f261cbf: Log matching property violated. Preceding OpId in replica: term: 2 index: 346. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
I20240502 21:26:44.118269 24054 consensus_queue.cc:1035] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 21:26:44.120549 24053 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf [term 2 LEADER]: Committing config change with OpId 2.347: config changed from index 345 to 347, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:44.121170 23493 raft_consensus.cc:2932] T 7875028620db4a8bbad7f0c5d1331f5e P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Committing config change with OpId 2.347: config changed from index 345 to 347, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } } }
I20240502 21:26:44.125578 20272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7875028620db4a8bbad7f0c5d1331f5e with cas_config_opid_index 345: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:44.126029 20281 catalog_manager.cc:5484] T 7875028620db4a8bbad7f0c5d1331f5e P 654f8d3e16bb4a658607a5c26f261cbf reported cstate change: config changed from index 345 to 347, VOTER 35812e2216cb4758aee1e94877a823b7 (127.16.122.1) evicted. New cstate: current_term: 2 leader_uuid: "654f8d3e16bb4a658607a5c26f261cbf" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "654f8d3e16bb4a658607a5c26f261cbf" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:44.129468 20408 tablet_service.cc:1514] Processing DeleteTablet for tablet 7875028620db4a8bbad7f0c5d1331f5e with delete_type TABLET_DATA_TOMBSTONED (TS 35812e2216cb4758aee1e94877a823b7 not found in new config with opid_index 347) from {username='slave'} at 127.0.0.1:34202
I20240502 21:26:44.130178 24055 tablet_replica.cc:330] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: stopping tablet replica
I20240502 21:26:44.130467 24055 raft_consensus.cc:2229] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.130741 24055 raft_consensus.cc:2258] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.132478 24055 ts_tablet_manager.cc:1875] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:44.136900 24055 ts_tablet_manager.cc:1888] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.346
I20240502 21:26:44.137017 24055 log.cc:1196] T 7875028620db4a8bbad7f0c5d1331f5e P 35812e2216cb4758aee1e94877a823b7: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/7875028620db4a8bbad7f0c5d1331f5e
I20240502 21:26:44.137805 20274 catalog_manager.cc:4830] TS 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): tablet 7875028620db4a8bbad7f0c5d1331f5e (table rebalance_test_table_1 [id=518f4738239746d6b8d9a481360bac0d]) successfully deleted
I20240502 21:26:44.306440 21733 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:44.308432 23474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:44.308940 20678 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:44.326478 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:44.343132 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:44.361234 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:44.407545 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:44.419757 21864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:44.600023 23474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:44.608304 20678 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:44.622129 21733 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:44.636335 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:44.655499 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:44.660629 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:44.679150 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:44.703899 21864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:45.000816 23474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:45.013406 20678 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:45.017022 21733 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:45.043224 20408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:45.044468 20544 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:45.044975 21994 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:45.069770 22127 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:45.090595 21864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ff826cbe705f48c5b303117140a0e6d2 | 127.16.122.62:40689 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.122.20:39533 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0230de442a8d4b9a968eb35462229e33 | 127.16.122.8:40417 | HEALTHY | <none> | 1 | 0
0ac8c8b961dc4500bc8b9f1055298244 | 127.16.122.3:34769 | HEALTHY | <none> | 3 | 0
16745bde14bc4a43b532e6bb828f0625 | 127.16.122.4:46571 | HEALTHY | <none> | 2 | 0
35812e2216cb4758aee1e94877a823b7 | 127.16.122.1:39539 | HEALTHY | <none> | 3 | 0
654f8d3e16bb4a658607a5c26f261cbf | 127.16.122.2:40149 | HEALTHY | <none> | 3 | 0
6c0aa2a3f325429eb83423bff3a80761 | 127.16.122.6:38855 | HEALTHY | <none> | 0 | 0
b4edc0e5b5c04ac5a7fe3f99d86d314b | 127.16.122.7:34365 | HEALTHY | <none> | 2 | 0
ee8dcd502996434ea55a16a62a5324fb | 127.16.122.5:35311 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.16.122.1 | experimental | 127.16.122.1:39539
local_ip_for_outbound_sockets | 127.16.122.2 | experimental | 127.16.122.2:40149
local_ip_for_outbound_sockets | 127.16.122.3 | experimental | 127.16.122.3:34769
local_ip_for_outbound_sockets | 127.16.122.4 | experimental | 127.16.122.4:46571
local_ip_for_outbound_sockets | 127.16.122.5 | experimental | 127.16.122.5:35311
local_ip_for_outbound_sockets | 127.16.122.6 | experimental | 127.16.122.6:38855
local_ip_for_outbound_sockets | 127.16.122.7 | experimental | 127.16.122.7:34365
local_ip_for_outbound_sockets | 127.16.122.8 | experimental | 127.16.122.8:40417
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.122.1:39539
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.122.2:40149
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.122.3:34769
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.122.4:46571
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.122.5:35311
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.122.6:38855
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.122.7:34365
server_dump_info_path | /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.16.122.8:40417
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.122.20:39533 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:26:45.153740 16872 log_verifier.cc:126] Checking tablet 028bf9972bf44bbbad3b862c70b174ca
I20240502 21:26:45.264259 16872 log_verifier.cc:177] Verified matching terms for 288 ops in tablet 028bf9972bf44bbbad3b862c70b174ca
I20240502 21:26:45.264694 16872 log_verifier.cc:126] Checking tablet 0e14655d22984ef3aab11be93eaaa07d
I20240502 21:26:45.276255 16872 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 0e14655d22984ef3aab11be93eaaa07d
I20240502 21:26:45.276362 16872 log_verifier.cc:126] Checking tablet 12887227115743afacb541c753db51c9
I20240502 21:26:45.402065 16872 log_verifier.cc:177] Verified matching terms for 329 ops in tablet 12887227115743afacb541c753db51c9
I20240502 21:26:45.402770 16872 log_verifier.cc:126] Checking tablet 129cc6d483b842fead09a5b1ddd58f52
I20240502 21:26:45.553898 16872 log_verifier.cc:177] Verified matching terms for 282 ops in tablet 129cc6d483b842fead09a5b1ddd58f52
I20240502 21:26:45.554304 16872 log_verifier.cc:126] Checking tablet 2998322d830948ac858235f4fb7defca
I20240502 21:26:45.689435 16872 log_verifier.cc:177] Verified matching terms for 329 ops in tablet 2998322d830948ac858235f4fb7defca
I20240502 21:26:45.690035 16872 log_verifier.cc:126] Checking tablet 380fd60c78334d90b26c3c9ce4945247
I20240502 21:26:45.815133 16872 log_verifier.cc:177] Verified matching terms for 289 ops in tablet 380fd60c78334d90b26c3c9ce4945247
I20240502 21:26:45.815537 16872 log_verifier.cc:126] Checking tablet 3fabe357cd7d42259e10511baa31b0bc
I20240502 21:26:45.940323 16872 log_verifier.cc:177] Verified matching terms for 281 ops in tablet 3fabe357cd7d42259e10511baa31b0bc
I20240502 21:26:45.940750 16872 log_verifier.cc:126] Checking tablet 44116b77039245b4883892cf19a8a248
I20240502 21:26:46.064567 16872 log_verifier.cc:177] Verified matching terms for 288 ops in tablet 44116b77039245b4883892cf19a8a248
I20240502 21:26:46.064962 16872 log_verifier.cc:126] Checking tablet 7875028620db4a8bbad7f0c5d1331f5e
I20240502 21:26:46.206473 16872 log_verifier.cc:177] Verified matching terms for 347 ops in tablet 7875028620db4a8bbad7f0c5d1331f5e
I20240502 21:26:46.206946 16872 log_verifier.cc:126] Checking tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f
I20240502 21:26:46.313517 16872 log_verifier.cc:177] Verified matching terms for 276 ops in tablet 8bdf4a8993904b02b2dd0fe3ba0f9e4f
I20240502 21:26:46.313967 16872 log_verifier.cc:126] Checking tablet 99a0d6a8a5a348dba1e8fd42d46b5f61
I20240502 21:26:46.414408 16872 log_verifier.cc:177] Verified matching terms for 261 ops in tablet 99a0d6a8a5a348dba1e8fd42d46b5f61
I20240502 21:26:46.414783 16872 log_verifier.cc:126] Checking tablet 9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:46.533340 16872 log_verifier.cc:177] Verified matching terms for 314 ops in tablet 9ca7cd6b72d4419aade387eaf375454e
I20240502 21:26:46.533776 16872 log_verifier.cc:126] Checking tablet d0892742cff44146ae2fb345d7e2bd3a
I20240502 21:26:46.651094 16872 log_verifier.cc:177] Verified matching terms for 307 ops in tablet d0892742cff44146ae2fb345d7e2bd3a
I20240502 21:26:46.651554 16872 log_verifier.cc:126] Checking tablet d15373bff59348b9a65b42053e8df389
I20240502 21:26:46.770953 16872 log_verifier.cc:177] Verified matching terms for 296 ops in tablet d15373bff59348b9a65b42053e8df389
I20240502 21:26:46.771376 16872 log_verifier.cc:126] Checking tablet d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:46.879554 16872 log_verifier.cc:177] Verified matching terms for 297 ops in tablet d4aeebd7828643ccab4aa62d4fd4d12a
I20240502 21:26:46.879952 16872 log_verifier.cc:126] Checking tablet ef774f42c72349d0aa9d490bb5dd8011
I20240502 21:26:46.991784 16872 log_verifier.cc:177] Verified matching terms for 303 ops in tablet ef774f42c72349d0aa9d490bb5dd8011
I20240502 21:26:46.999662 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 20347
I20240502 21:26:47.001050 20469 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:47.231117 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 20347
W20240502 21:26:47.271998 20272 connection.cc:535] server connection from 127.16.122.1:40847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:47.272320 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 20479
I20240502 21:26:47.273380 20606 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:26:47.485445 21689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.1:39539: connect: Connection refused (error 111)
W20240502 21:26:47.487099 21689 consensus_peers.cc:485] T 380fd60c78334d90b26c3c9ce4945247 P 16745bde14bc4a43b532e6bb828f0625 -> Peer 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): Couldn't send request to peer 35812e2216cb4758aee1e94877a823b7. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.1:39539: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:47.518369 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 20479
W20240502 21:26:47.561208 21818 connection.cc:535] server connection from 127.16.122.2:36801 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:26:47.561249 21950 connection.cc:535] server connection from 127.16.122.2:54747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:26:47.561833 23426 connection.cc:535] server connection from 127.16.122.2:55035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:47.561998 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 20616
I20240502 21:26:47.563170 20739 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:47.808972 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 20616
W20240502 21:26:47.850390 22079 connection.cc:535] server connection from 127.16.122.3:59675 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:26:47.850735 21948 connection.cc:535] server connection from 127.16.122.3:41757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:47.851212 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 21671
I20240502 21:26:47.858376 21793 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:26:47.908586 23427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.3:34769: connect: Connection refused (error 111)
W20240502 21:26:47.910027 23427 consensus_peers.cc:485] T d4aeebd7828643ccab4aa62d4fd4d12a P 0230de442a8d4b9a968eb35462229e33 -> Peer 0ac8c8b961dc4500bc8b9f1055298244 (127.16.122.3:34769): Couldn't send request to peer 0ac8c8b961dc4500bc8b9f1055298244. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.3:34769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:48.040820 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 21671
I20240502 21:26:48.072990 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 21802
I20240502 21:26:48.079926 21923 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:48.268051 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 21802
I20240502 21:26:48.301139 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 21932
I20240502 21:26:48.307322 22054 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:48.337226 24183 raft_consensus.cc:482] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 35812e2216cb4758aee1e94877a823b7)
I20240502 21:26:48.337486 24183 raft_consensus.cc:504] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "35812e2216cb4758aee1e94877a823b7" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 39539 } } peers { permanent_uuid: "0230de442a8d4b9a968eb35462229e33" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 40417 } attrs { promote: false } }
I20240502 21:26:48.338276 24183 leader_election.cc:288] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539)
W20240502 21:26:48.339836 23428 leader_election.cc:334] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 35812e2216cb4758aee1e94877a823b7 (127.16.122.1:39539): Network error: Client connection negotiation failed: client connection to 127.16.122.1:39539: connect: Connection refused (error 111)
I20240502 21:26:48.339948 23428 leader_election.cc:302] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0230de442a8d4b9a968eb35462229e33; no voters: 35812e2216cb4758aee1e94877a823b7
I20240502 21:26:48.340229 24183 raft_consensus.cc:2726] T 99a0d6a8a5a348dba1e8fd42d46b5f61 P 0230de442a8d4b9a968eb35462229e33 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:26:48.464637 24203 raft_consensus.cc:482] T d0892742cff44146ae2fb345d7e2bd3a P 6c0aa2a3f325429eb83423bff3a80761 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 35812e2216cb4758aee1e94877a823b7)
I20240502 21:26:48.464859 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 21932
W20240502 21:26:48.493322 20273 connection.cc:535] server connection from 127.16.122.6:59739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:48.493604 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 22064
I20240502 21:26:48.500253 22189 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:48.663056 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 22064
I20240502 21:26:48.691712 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 23409
I20240502 21:26:48.698653 23534 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:48.826128 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 23409
I20240502 21:26:48.849380 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 20256
I20240502 21:26:48.850673 20317 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:49.027014 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 20256
2024-05-02T21:26:49Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (49331 ms)
[----------] 1 test from RebalanceParamTest (49331 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 21:26:49.123332 16872 test_util.cc:268] Using random seed: 1120573922
I20240502 21:26:49.125133 16872 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:26:49.125200 16872 ts_itest-base.cc:113] --------------
I20240502 21:26:49.125247 16872 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:26:49.125289 16872 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:26:49.125331 16872 ts_itest-base.cc:116] --------------
2024-05-02T21:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:26:49Z Disabled control of system clock
I20240502 21:26:49.178856 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:45825
--webserver_interface=127.16.122.62
--webserver_port=0
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.62:45825
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:26:49.551563 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:49.551937 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:49.552080 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:49.559907 24217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:49.560089 24217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:49.560191 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:49.560283 24217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:49.560355 24217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:49.569878 24217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.122.62:45825
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.62:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.122.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:49.571382 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:49.573228 24217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:49.584627 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:49.584734 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:49.585342 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:49.585976 24217 server_base.cc:1034] running on GCE node
I20240502 21:26:49.586815 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:49.588297 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:49.589509 24217 hybrid_clock.cc:648] HybridClock initialized: now 1714685209589456 us; error 48 us; skew 500 ppm
I20240502 21:26:49.589876 24217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:49.592330 24217 webserver.cc:460] Webserver started at http://127.16.122.62:45023/ using document root <none> and password file <none>
I20240502 21:26:49.593117 24217 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:49.593240 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:49.593595 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:49.596432 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "34199807c207492781e8dba15a3f67ab"
format_stamp: "Formatted at 2024-05-02 21:26:49 on dist-test-slave-p9xn"
I20240502 21:26:49.597352 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "34199807c207492781e8dba15a3f67ab"
format_stamp: "Formatted at 2024-05-02 21:26:49 on dist-test-slave-p9xn"
I20240502 21:26:49.602937 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:26:49.606511 24231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:49.607869 24217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:49.608049 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
uuid: "34199807c207492781e8dba15a3f67ab"
format_stamp: "Formatted at 2024-05-02 21:26:49 on dist-test-slave-p9xn"
I20240502 21:26:49.608258 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:49.620956 24217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:49.621811 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:49.622136 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:49.641053 24217 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.62:45825
I20240502 21:26:49.641075 24282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.62:45825 every 8 connection(s)
I20240502 21:26:49.642619 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:26:49.646556 24283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.652891 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24217
I20240502 21:26:49.653081 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:26:49.656502 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab: Bootstrap starting.
I20240502 21:26:49.660262 24283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.661532 24283 log.cc:824] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:49.664853 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab: No bootstrap required, opened a new log
I20240502 21:26:49.671684 24283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34199807c207492781e8dba15a3f67ab" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45825 } }
I20240502 21:26:49.672116 24283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.672238 24283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34199807c207492781e8dba15a3f67ab, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.672986 24283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34199807c207492781e8dba15a3f67ab" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45825 } }
I20240502 21:26:49.673254 24283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:49.673370 24283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:49.673498 24283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.675704 24283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34199807c207492781e8dba15a3f67ab" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45825 } }
I20240502 21:26:49.676199 24283 leader_election.cc:302] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34199807c207492781e8dba15a3f67ab; no voters:
I20240502 21:26:49.676832 24283 leader_election.cc:288] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:49.676981 24288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.678390 24288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [term 1 LEADER]: Becoming Leader. State: Replica: 34199807c207492781e8dba15a3f67ab, State: Running, Role: LEADER
I20240502 21:26:49.679075 24283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:49.678974 24288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34199807c207492781e8dba15a3f67ab" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45825 } }
I20240502 21:26:49.682240 24290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34199807c207492781e8dba15a3f67ab. Latest consensus state: current_term: 1 leader_uuid: "34199807c207492781e8dba15a3f67ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34199807c207492781e8dba15a3f67ab" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45825 } } }
I20240502 21:26:49.682719 24290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:49.682265 24289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34199807c207492781e8dba15a3f67ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34199807c207492781e8dba15a3f67ab" member_type: VOTER last_known_addr { host: "127.16.122.62" port: 45825 } } }
I20240502 21:26:49.683013 24289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:49.687587 24296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:49.692931 24296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:49.702414 24296 catalog_manager.cc:1347] Generated new cluster ID: 9eb988cb2c1447b8ada11391419f0e4f
I20240502 21:26:49.702556 24296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:49.715847 24296 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:49.717612 24296 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:49.734125 24296 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 34199807c207492781e8dba15a3f67ab: Generated new TSK 0
I20240502 21:26:49.735335 24296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:49.744658 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.1:0
--local_ip_for_outbound_sockets=127.16.122.1
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:26:50.117262 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:50.117584 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:50.117678 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:50.125811 24307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:50.125953 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:50.126087 24307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.1
I20240502 21:26:50.135597 24307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.122.1
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.1
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:50.136997 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:50.139159 24307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:50.152482 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:50.152554 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:50.153039 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:50.153712 24307 server_base.cc:1034] running on GCE node
I20240502 21:26:50.154376 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:50.155346 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:50.156549 24307 hybrid_clock.cc:648] HybridClock initialized: now 1714685210156511 us; error 34 us; skew 500 ppm
I20240502 21:26:50.156865 24307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:50.159211 24307 webserver.cc:460] Webserver started at http://127.16.122.1:35415/ using document root <none> and password file <none>
I20240502 21:26:50.159979 24307 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:50.160097 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:50.160434 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:50.163211 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e856847c3f494e6aa57f9cecf117ce4d"
format_stamp: "Formatted at 2024-05-02 21:26:50 on dist-test-slave-p9xn"
I20240502 21:26:50.163983 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e856847c3f494e6aa57f9cecf117ce4d"
format_stamp: "Formatted at 2024-05-02 21:26:50 on dist-test-slave-p9xn"
I20240502 21:26:50.169606 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:26:50.173168 24321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:50.174636 24307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:50.174815 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e856847c3f494e6aa57f9cecf117ce4d"
format_stamp: "Formatted at 2024-05-02 21:26:50 on dist-test-slave-p9xn"
I20240502 21:26:50.175020 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:50.195294 24307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:50.196225 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:50.196530 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:50.198163 24307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:50.200287 24307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:50.200384 24307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:50.200474 24307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:50.200529 24307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:50.233968 24307 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.1:36473
I20240502 21:26:50.234009 24435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.1:36473 every 8 connection(s)
I20240502 21:26:50.235342 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:50.242472 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24307
I20240502 21:26:50.242666 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:26:50.246166 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.2:0
--local_ip_for_outbound_sockets=127.16.122.2
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:50.248553 24436 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:50.248857 24436 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:50.249686 24436 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:50.251753 24248 ts_manager.cc:194] Registered new tserver with Master: e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:26:50.253268 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.1:36353
W20240502 21:26:50.596800 24440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:50.597175 24440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:50.597281 24440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:50.605105 24440 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:50.605242 24440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:50.605371 24440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.2
I20240502 21:26:50.614897 24440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.122.2
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.2
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:50.616240 24440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:50.618119 24440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:50.631140 24448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:50.631211 24446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:50.631461 24445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:50.632088 24440 server_base.cc:1034] running on GCE node
I20240502 21:26:50.632678 24440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:50.633631 24440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:50.634835 24440 hybrid_clock.cc:648] HybridClock initialized: now 1714685210634779 us; error 46 us; skew 500 ppm
I20240502 21:26:50.635160 24440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:50.637455 24440 webserver.cc:460] Webserver started at http://127.16.122.2:45255/ using document root <none> and password file <none>
I20240502 21:26:50.638223 24440 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:50.638374 24440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:50.638711 24440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:50.641386 24440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7f8bb40905da4891b461220b95c66832"
format_stamp: "Formatted at 2024-05-02 21:26:50 on dist-test-slave-p9xn"
I20240502 21:26:50.642158 24440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7f8bb40905da4891b461220b95c66832"
format_stamp: "Formatted at 2024-05-02 21:26:50 on dist-test-slave-p9xn"
I20240502 21:26:50.647442 24440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 21:26:50.651018 24454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:50.652410 24440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:50.652570 24440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7f8bb40905da4891b461220b95c66832"
format_stamp: "Formatted at 2024-05-02 21:26:50 on dist-test-slave-p9xn"
I20240502 21:26:50.652782 24440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:50.681329 24440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:50.682241 24440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:50.682556 24440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:50.683566 24440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:50.685087 24440 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:50.685175 24440 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:50.685295 24440 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:50.685377 24440 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:50.720314 24440 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.2:43159
I20240502 21:26:50.720388 24566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.2:43159 every 8 connection(s)
I20240502 21:26:50.721675 24440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:26:50.729751 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24440
I20240502 21:26:50.729897 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:26:50.733389 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.3:0
--local_ip_for_outbound_sockets=127.16.122.3
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:50.735471 24567 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:50.735814 24567 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:50.736696 24567 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:50.738291 24248 ts_manager.cc:194] Registered new tserver with Master: 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:26:50.739455 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.2:51399
W20240502 21:26:51.083325 24571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:51.083647 24571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:51.083791 24571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:51.091624 24571 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:51.091768 24571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:51.091964 24571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.3
I20240502 21:26:51.101817 24571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.122.3
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.3
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:51.103286 24571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:51.105155 24571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:51.117574 24576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.117611 24579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.118198 24577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:51.118937 24571 server_base.cc:1034] running on GCE node
I20240502 21:26:51.119554 24571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:51.120528 24571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:51.121726 24571 hybrid_clock.cc:648] HybridClock initialized: now 1714685211121676 us; error 40 us; skew 500 ppm
I20240502 21:26:51.122048 24571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:51.124457 24571 webserver.cc:460] Webserver started at http://127.16.122.3:44377/ using document root <none> and password file <none>
I20240502 21:26:51.125232 24571 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:51.125346 24571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:51.125634 24571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:51.128502 24571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "427345e9a65d417c84a773ac1925a18a"
format_stamp: "Formatted at 2024-05-02 21:26:51 on dist-test-slave-p9xn"
I20240502 21:26:51.129254 24571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "427345e9a65d417c84a773ac1925a18a"
format_stamp: "Formatted at 2024-05-02 21:26:51 on dist-test-slave-p9xn"
I20240502 21:26:51.134613 24571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 21:26:51.138005 24585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.139369 24571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:51.139530 24571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "427345e9a65d417c84a773ac1925a18a"
format_stamp: "Formatted at 2024-05-02 21:26:51 on dist-test-slave-p9xn"
I20240502 21:26:51.139732 24571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:51.155898 24571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:51.156752 24571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:51.157023 24571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:51.158115 24571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:51.159911 24571 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:51.160006 24571 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.160122 24571 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:51.160192 24571 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.194096 24571 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.3:38427
I20240502 21:26:51.194180 24697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.3:38427 every 8 connection(s)
I20240502 21:26:51.195531 24571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:26:51.197779 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24571
I20240502 21:26:51.197945 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:26:51.201997 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.4:0
--local_ip_for_outbound_sockets=127.16.122.4
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:51.210904 24698 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:51.211270 24698 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:51.212414 24698 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:51.214185 24248 ts_manager.cc:194] Registered new tserver with Master: 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:51.215210 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.3:35805
I20240502 21:26:51.256206 24436 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
W20240502 21:26:51.554495 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:51.554824 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:51.554944 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:51.562762 24701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:51.562896 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:51.563061 24701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.4
I20240502 21:26:51.572535 24701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.122.4
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.4
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:51.573989 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:51.575896 24701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:51.588506 24708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.588596 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:51.589154 24710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:51.589843 24701 server_base.cc:1034] running on GCE node
I20240502 21:26:51.590552 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:51.591563 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:51.592782 24701 hybrid_clock.cc:648] HybridClock initialized: now 1714685211592738 us; error 41 us; skew 500 ppm
I20240502 21:26:51.593123 24701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:51.595618 24701 webserver.cc:460] Webserver started at http://127.16.122.4:37807/ using document root <none> and password file <none>
I20240502 21:26:51.596414 24701 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:51.596534 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:51.596853 24701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:51.599653 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1398472b92c842f285b8c7cc43cc1527"
format_stamp: "Formatted at 2024-05-02 21:26:51 on dist-test-slave-p9xn"
I20240502 21:26:51.600472 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1398472b92c842f285b8c7cc43cc1527"
format_stamp: "Formatted at 2024-05-02 21:26:51 on dist-test-slave-p9xn"
I20240502 21:26:51.605980 24701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:26:51.609618 24716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.611074 24701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:51.611248 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1398472b92c842f285b8c7cc43cc1527"
format_stamp: "Formatted at 2024-05-02 21:26:51 on dist-test-slave-p9xn"
I20240502 21:26:51.611464 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:51.638136 24701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:51.639218 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:51.639505 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:51.640527 24701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:51.642519 24701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:51.642627 24701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.642731 24701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:51.642799 24701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:51.682910 24701 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.4:33687
I20240502 21:26:51.682968 24828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.4:33687 every 8 connection(s)
I20240502 21:26:51.684247 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:51.687615 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24701
I20240502 21:26:51.687806 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:26:51.691351 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.5:0
--local_ip_for_outbound_sockets=127.16.122.5
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:51.698189 24829 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:51.698643 24829 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:51.699810 24829 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:51.701678 24248 ts_manager.cc:194] Registered new tserver with Master: 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:26:51.702874 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.4:45025
I20240502 21:26:51.742180 24567 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
W20240502 21:26:52.048620 24832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:52.048915 24832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:52.049047 24832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:52.056687 24832 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:52.056828 24832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:52.057034 24832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.5
I20240502 21:26:52.067190 24832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.122.5
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.5
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:52.068707 24832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:52.070726 24832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:52.084313 24838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:52.084609 24840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:52.084867 24837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:52.085645 24832 server_base.cc:1034] running on GCE node
I20240502 21:26:52.086325 24832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:52.087441 24832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:52.088672 24832 hybrid_clock.cc:648] HybridClock initialized: now 1714685212088623 us; error 39 us; skew 500 ppm
I20240502 21:26:52.089040 24832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:52.091547 24832 webserver.cc:460] Webserver started at http://127.16.122.5:36891/ using document root <none> and password file <none>
I20240502 21:26:52.092392 24832 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:52.092502 24832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:52.092845 24832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:52.095732 24832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b77f8f52372241d483f457389fda0f93"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:26:52.096587 24832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b77f8f52372241d483f457389fda0f93"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:26:52.102236 24832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 21:26:52.106022 24846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:52.107416 24832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:52.107587 24832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b77f8f52372241d483f457389fda0f93"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:26:52.107801 24832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:52.140569 24832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:52.141518 24832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:52.141820 24832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:52.142871 24832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:52.144608 24832 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:52.144708 24832 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:52.144829 24832 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:52.144901 24832 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:52.177563 24832 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.5:42587
I20240502 21:26:52.177616 24959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.5:42587 every 8 connection(s)
I20240502 21:26:52.179080 24832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:52.188242 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24832
I20240502 21:26:52.188469 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:26:52.192695 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.6:0
--local_ip_for_outbound_sockets=127.16.122.6
--webserver_interface=127.16.122.6
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:52.193147 24960 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:52.193504 24960 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:52.194672 24960 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:52.196640 24248 ts_manager.cc:194] Registered new tserver with Master: b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:26:52.197821 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.5:60281
I20240502 21:26:52.217806 24698 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
W20240502 21:26:52.543752 24963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:52.544107 24963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:52.544245 24963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:52.552145 24963 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:52.552301 24963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:52.552476 24963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.6
I20240502 21:26:52.561966 24963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.122.6
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.6
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:52.563441 24963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:52.565212 24963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:52.577445 24969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:52.577780 24971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:52.577950 24968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:52.578586 24963 server_base.cc:1034] running on GCE node
I20240502 21:26:52.579288 24963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:52.580293 24963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:52.581501 24963 hybrid_clock.cc:648] HybridClock initialized: now 1714685212581445 us; error 60 us; skew 500 ppm
I20240502 21:26:52.581805 24963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:52.584053 24963 webserver.cc:460] Webserver started at http://127.16.122.6:36403/ using document root <none> and password file <none>
I20240502 21:26:52.584803 24963 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:52.584919 24963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:52.585242 24963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:52.587961 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "836775e329f148e5908a1afc54f06428"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:26:52.588761 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "836775e329f148e5908a1afc54f06428"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:26:52.594025 24963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:26:52.597496 24977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:52.599004 24963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:52.599164 24963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "836775e329f148e5908a1afc54f06428"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:26:52.599354 24963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:52.618180 24963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:52.619074 24963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:52.619354 24963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:52.620374 24963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:52.621874 24963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:52.621971 24963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:52.622090 24963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:52.622166 24963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:52.654498 24963 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.6:41751
I20240502 21:26:52.654531 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.6:41751 every 8 connection(s)
I20240502 21:26:52.655939 24963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:52.658243 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 24963
I20240502 21:26:52.658433 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:26:52.662220 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.7:0
--local_ip_for_outbound_sockets=127.16.122.7
--webserver_interface=127.16.122.7
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:52.670996 25091 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:52.671383 25091 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:52.672498 25091 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:52.674176 24248 ts_manager.cc:194] Registered new tserver with Master: 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751)
I20240502 21:26:52.675276 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.6:43809
I20240502 21:26:52.705605 24829 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
W20240502 21:26:53.016786 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:53.017154 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:53.017294 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:53.025621 25094 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:53.025781 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:53.025981 25094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.7
I20240502 21:26:53.035753 25094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.122.7
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.7
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:53.037209 25094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:53.039098 25094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:53.052407 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.052465 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.053200 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:53.053609 25094 server_base.cc:1034] running on GCE node
I20240502 21:26:53.054266 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:53.055362 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:53.056627 25094 hybrid_clock.cc:648] HybridClock initialized: now 1714685213056559 us; error 47 us; skew 500 ppm
I20240502 21:26:53.057053 25094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:53.059728 25094 webserver.cc:460] Webserver started at http://127.16.122.7:39651/ using document root <none> and password file <none>
I20240502 21:26:53.060555 25094 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:53.060680 25094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:53.061021 25094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:53.063848 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9b0952b24bd8400c9940dbd5e28417b4"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:26:53.064648 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9b0952b24bd8400c9940dbd5e28417b4"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:26:53.070269 25094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:26:53.074154 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:53.075973 25094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 21:26:53.076169 25094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9b0952b24bd8400c9940dbd5e28417b4"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:26:53.076442 25094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:53.092145 25094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:53.093158 25094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:53.093482 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:53.094651 25094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:53.096315 25094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:53.096429 25094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:53.096552 25094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:53.096629 25094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:53.131050 25094 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.7:36311
I20240502 21:26:53.131095 25221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.7:36311 every 8 connection(s)
I20240502 21:26:53.132493 25094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:53.138320 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 25094
I20240502 21:26:53.138552 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:26:53.141749 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.8:0
--local_ip_for_outbound_sockets=127.16.122.8
--webserver_interface=127.16.122.8
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:53.146448 25222 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:53.146888 25222 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:53.148038 25222 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:53.149906 24248 ts_manager.cc:194] Registered new tserver with Master: 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
I20240502 21:26:53.150786 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.7:48761
I20240502 21:26:53.201272 24960 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
W20240502 21:26:53.502539 25225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:53.502854 25225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:53.502948 25225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:53.510731 25225 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:53.510867 25225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:53.511000 25225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.8
I20240502 21:26:53.520665 25225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.122.8
--webserver_port=0
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.8
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:53.522014 25225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:53.523901 25225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:53.536617 25231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.536952 25230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:53.537175 25233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:53.537760 25225 server_base.cc:1034] running on GCE node
I20240502 21:26:53.538532 25225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:53.539570 25225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:53.540809 25225 hybrid_clock.cc:648] HybridClock initialized: now 1714685213540763 us; error 38 us; skew 500 ppm
I20240502 21:26:53.541157 25225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:53.543726 25225 webserver.cc:460] Webserver started at http://127.16.122.8:34171/ using document root <none> and password file <none>
I20240502 21:26:53.544498 25225 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:53.544620 25225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:53.544956 25225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:53.547734 25225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1769ee000eb24298a367a0de1209e6e0"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:26:53.548554 25225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1769ee000eb24298a367a0de1209e6e0"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:26:53.554242 25225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 21:26:53.558043 25239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:53.559574 25225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 21:26:53.559749 25225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1769ee000eb24298a367a0de1209e6e0"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:26:53.559955 25225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:53.581342 25225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:53.582267 25225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:53.582574 25225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:53.583572 25225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:53.585059 25225 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:53.585155 25225 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:53.585275 25225 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:53.585354 25225 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:53.619462 25225 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.8:37045
I20240502 21:26:53.619521 25352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.8:37045 every 8 connection(s)
I20240502 21:26:53.620774 25225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:53.627457 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 25225
I20240502 21:26:53.627663 16872 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:26:53.634269 25353 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:26:53.634609 25353 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:53.635411 25353 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:26:53.636723 24248 ts_manager.cc:194] Registered new tserver with Master: 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045)
I20240502 21:26:53.637543 24248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.8:40185
I20240502 21:26:53.644361 16872 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:26:53.661901 16872 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:26:53.662045 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 24832
I20240502 21:26:53.670226 24955 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:53.677929 25091 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
I20240502 21:26:53.754647 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 24832
I20240502 21:26:53.771844 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 24963
I20240502 21:26:53.780651 25086 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:53.863332 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 24963
I20240502 21:26:53.880959 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 25094
I20240502 21:26:53.889711 25217 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:53.972422 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 25094
I20240502 21:26:53.989580 16872 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu pid 25225
I20240502 21:26:53.999856 25348 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:54.084787 16872 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu with pid 25225
I20240502 21:26:57.854013 24248 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:57.897125 24368 tablet_service.cc:1467] Processing CreateTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.896945 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.895892 24369 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.898478 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b080037646b4b8b34c0eab2de4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.898969 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.900985 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.914963 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.916585 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.919711 24763 tablet_service.cc:1467] Processing CreateTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.920634 25382 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:26:57.921761 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.922526 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b080037646b4b8b34c0eab2de4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.923553 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.924289 24501 tablet_service.cc:1467] Processing CreateTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.924862 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b080037646b4b8b34c0eab2de4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.928790 25382 tablet_bootstrap.cc:654] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.930148 25382 log.cc:824] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:57.933399 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.933440 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.935053 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.935564 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.944757 25384 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:26:57.948689 25384 tablet_bootstrap.cc:654] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.950446 25384 log.cc:824] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:57.951485 25385 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:26:57.956101 25385 tablet_bootstrap.cc:654] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.957703 25385 log.cc:824] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:57.960498 25384 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:26:57.960902 25384 ts_tablet_manager.cc:1367] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240502 21:26:57.962646 25387 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:26:57.963369 25382 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:26:57.963739 25382 ts_tablet_manager.cc:1367] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.011s
I20240502 21:26:57.966315 25387 tablet_bootstrap.cc:654] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.967984 25387 log.cc:824] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:57.973435 25384 raft_consensus.cc:348] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:57.974090 25384 raft_consensus.cc:374] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.974238 25384 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.975631 25384 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:57.976588 25382 raft_consensus.cc:348] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:57.977133 25382 raft_consensus.cc:374] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.977281 25382 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.977792 25385 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:26:57.978119 25385 ts_tablet_manager.cc:1367] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.008s
I20240502 21:26:57.978233 25384 ts_tablet_manager.cc:1398] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20240502 21:26:57.978165 25382 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:57.978669 25387 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
W20240502 21:26:57.979022 24569 tablet.cc:2369] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:57.979190 25384 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:26:57.981241 25384 tablet_bootstrap.cc:654] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.987387 25387 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.003s
I20240502 21:26:57.989230 25385 raft_consensus.cc:348] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:57.989799 25385 raft_consensus.cc:374] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.989950 25385 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.991014 25385 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:57.993732 25385 ts_tablet_manager.cc:1398] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20240502 21:26:57.994182 25385 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:26:57.994560 25384 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:26:57.994728 25384 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20240502 21:26:57.995904 25385 tablet_bootstrap.cc:654] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.995827 25384 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:57.996042 25384 raft_consensus.cc:374] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.996137 25384 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.996385 25384 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:57.996994 25384 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:58.003707 25387 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.004129 25387 raft_consensus.cc:374] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.004241 25387 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.004912 25387 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.008566 25387 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.021s user 0.011s sys 0.001s
I20240502 21:26:58.008739 25382 ts_tablet_manager.cc:1398] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.045s user 0.020s sys 0.015s
I20240502 21:26:58.008993 25387 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:26:58.010380 25382 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:26:58.010761 25387 tablet_bootstrap.cc:654] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.012503 25382 tablet_bootstrap.cc:654] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.013126 25385 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:26:58.013355 25385 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240502 21:26:58.014256 25387 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:26:58.014477 25387 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:26:58.014484 25385 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.014729 25385 raft_consensus.cc:374] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.014837 25385 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.015089 25385 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.015880 25385 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:58.016032 25387 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.016353 25387 raft_consensus.cc:374] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.016467 25387 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.016805 25387 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.017606 25387 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:58.019132 25382 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:26:58.019297 25382 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240502 21:26:58.020112 25382 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.020295 25382 raft_consensus.cc:374] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.020401 25382 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.020692 25382 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.021122 25382 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:58.021364 25382 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:26:58.022655 25382 tablet_bootstrap.cc:654] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.024596 25382 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:26:58.024760 25382 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:58.025337 25382 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.025527 25382 raft_consensus.cc:374] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.025607 25382 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.025830 25382 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.026392 25382 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:58.115368 25391 raft_consensus.cc:482] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.115689 25391 raft_consensus.cc:504] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.117623 25391 leader_election.cc:288] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:58.126507 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf18ffb96d04d5e9ec8ee9563812ba0" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:26:58.126523 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf18ffb96d04d5e9ec8ee9563812ba0" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:26:58.127002 24653 raft_consensus.cc:2454] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:26:58.127005 24784 raft_consensus.cc:2454] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:26:58.127636 24324 leader_election.cc:302] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:26:58.128034 25391 raft_consensus.cc:2781] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.128196 25391 raft_consensus.cc:482] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.128341 25391 raft_consensus.cc:3037] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.130491 25391 raft_consensus.cc:504] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.131131 25391 leader_election.cc:288] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:58.131665 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf18ffb96d04d5e9ec8ee9563812ba0" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:26:58.131701 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf18ffb96d04d5e9ec8ee9563812ba0" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:26:58.131901 24653 raft_consensus.cc:3037] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.131906 24784 raft_consensus.cc:3037] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.135154 24784 raft_consensus.cc:2454] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:26:58.135154 24653 raft_consensus.cc:2454] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:26:58.135653 24324 leader_election.cc:302] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:26:58.135970 25391 raft_consensus.cc:2781] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.136389 25391 raft_consensus.cc:686] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:26:58.136921 25391 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.141791 24248 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:58.191504 24830 tablet.cc:2369] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:58.192256 25390 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.192523 25390 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:58.194391 25390 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
W20240502 21:26:58.202556 24699 tablet.cc:2369] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:58.202522 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:26:58.202818 24784 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:26:58.203743 24456 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:26:58.203823 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:26:58.204262 24390 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:26:58.204257 25390 raft_consensus.cc:2781] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.204413 25390 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.204520 25390 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.207643 25390 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:58.208353 25390 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:26:58.208782 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:26:58.208873 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:26:58.208984 24784 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.209089 24390 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.211108 24784 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:26:58.211297 24390 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:26:58.211613 24456 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:26:58.211910 25390 raft_consensus.cc:2781] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.212368 25390 raft_consensus.cc:686] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:26:58.212888 25390 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:26:58.218111 24248 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:58.244242 24437 tablet.cc:2369] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:58.263887 25390 raft_consensus.cc:482] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.264144 25390 raft_consensus.cc:504] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.264832 25390 leader_election.cc:288] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:58.265451 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:26:58.265769 24390 raft_consensus.cc:2454] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:26:58.266214 24456 leader_election.cc:302] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:26:58.266515 25390 raft_consensus.cc:2781] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.266593 25390 raft_consensus.cc:482] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.266697 25390 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.268705 25390 raft_consensus.cc:504] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.269366 25390 leader_election.cc:288] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:58.269812 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:26:58.270061 24390 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.271024 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:26:58.271077 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:26:58.271306 24653 raft_consensus.cc:2454] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:26:58.272580 24390 raft_consensus.cc:2454] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:26:58.272908 24456 leader_election.cc:302] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters: 427345e9a65d417c84a773ac1925a18a
I20240502 21:26:58.273180 25390 raft_consensus.cc:2781] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.273393 25390 raft_consensus.cc:686] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:26:58.273651 25390 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:58.276224 24248 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.579741 25391 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.590973 25401 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.652314 25390 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.652889 24652 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.657387 25390 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.698312 25405 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.705338 25390 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:58.736845 25408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:58.737293 25408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:58.746389 25408 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 249.217 ms
time per row: 0.124609 ms
I20240502 21:26:59.156093 24239 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:59.165380 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.165920 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.165805 24763 tablet_service.cc:1467] Processing CreateTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.166270 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.166241 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5a87719c2544a5b0cb591ed9f46df8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.166661 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5a87719c2544a5b0cb591ed9f46df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.167237 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5a87719c2544a5b0cb591ed9f46df8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.167663 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5a87719c2544a5b0cb591ed9f46df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.171241 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.171679 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.176012 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5a87719c2544a5b0cb591ed9f46df8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.176012 24631 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8a25580bc74004b0e31c34f841d2cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.176620 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5a87719c2544a5b0cb591ed9f46df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.176621 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.179039 24368 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8a25580bc74004b0e31c34f841d2cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.179668 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.180877 25479 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:26:59.183717 24500 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8a25580bc74004b0e31c34f841d2cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.184266 24500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.184883 25479 tablet_bootstrap.cc:654] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.185562 25481 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:26:59.187610 25481 tablet_bootstrap.cc:654] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.198235 25479 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:26:59.198486 25479 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.003s
I20240502 21:26:59.199396 25479 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.199627 25479 raft_consensus.cc:374] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.199702 25479 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.199996 25479 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.201417 25479 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.201682 25479 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:26:59.203966 25479 tablet_bootstrap.cc:654] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.212718 25479 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:26:59.212913 25479 ts_tablet_manager.cc:1367] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 21:26:59.213732 25479 raft_consensus.cc:348] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.213991 25479 raft_consensus.cc:374] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.214087 25479 raft_consensus.cc:729] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.214386 25479 consensus_queue.cc:260] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.215572 25479 ts_tablet_manager.cc:1398] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:26:59.218912 25485 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:26:59.221576 25485 tablet_bootstrap.cc:654] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.223408 25480 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:26:59.224696 25481 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:26:59.225018 25481 ts_tablet_manager.cc:1367] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.017s
I20240502 21:26:59.225607 25480 tablet_bootstrap.cc:654] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.226152 25481 raft_consensus.cc:348] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.226459 25481 raft_consensus.cc:374] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.226558 25481 raft_consensus.cc:729] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.226933 25481 consensus_queue.cc:260] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.227872 25481 ts_tablet_manager.cc:1398] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.228253 25481 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:26:59.229969 25481 tablet_bootstrap.cc:654] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.235908 25485 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:26:59.236117 25485 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20240502 21:26:59.237335 25485 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.237649 25485 raft_consensus.cc:374] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.237792 25485 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.238171 25485 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.239039 25485 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:59.239961 25481 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:26:59.240156 25481 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20240502 21:26:59.241216 25481 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.241508 25481 raft_consensus.cc:374] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.241607 25481 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.242031 25481 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.242877 25481 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:59.243266 25481 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:26:59.246112 25481 tablet_bootstrap.cc:654] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.251104 25480 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:26:59.251271 25480 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.013s
I20240502 21:26:59.252099 25481 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:26:59.252030 25480 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.252197 25480 raft_consensus.cc:374] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.252238 25481 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240502 21:26:59.252292 25480 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.252593 25480 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.253224 25481 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.253402 25481 raft_consensus.cc:374] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.253472 25481 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.253693 25481 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.254864 25481 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:59.256676 25480 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240502 21:26:59.257072 25480 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:26:59.258944 25480 tablet_bootstrap.cc:654] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.261794 25480 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:26:59.261940 25480 ts_tablet_manager.cc:1367] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:59.262566 25480 raft_consensus.cc:348] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.262748 25480 raft_consensus.cc:374] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.262832 25480 raft_consensus.cc:729] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.263074 25480 consensus_queue.cc:260] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.263567 25480 ts_tablet_manager.cc:1398] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:59.263867 25480 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:26:59.265110 25480 tablet_bootstrap.cc:654] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.268020 25480 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:26:59.268141 25480 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:59.268780 25480 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.268937 25480 raft_consensus.cc:374] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.269023 25480 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.269263 25480 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.269778 25480 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:59.295547 25401 raft_consensus.cc:482] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.295800 25401 raft_consensus.cc:504] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.296711 25401 leader_election.cc:288] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:59.297444 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8a25580bc74004b0e31c34f841d2cc" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:26:59.297720 24652 raft_consensus.cc:2454] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:26:59.298249 24324 leader_election.cc:302] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:26:59.298476 25454 raft_consensus.cc:482] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.298646 25401 raft_consensus.cc:2781] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.298774 25401 raft_consensus.cc:482] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.298671 25454 raft_consensus.cc:504] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.298903 25401 raft_consensus.cc:3037] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.299501 25454 leader_election.cc:288] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:26:59.300374 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5a87719c2544a5b0cb591ed9f46df8" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:26:59.300613 24784 raft_consensus.cc:2454] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:26:59.300793 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5a87719c2544a5b0cb591ed9f46df8" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:26:59.300992 24652 raft_consensus.cc:2454] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:26:59.301144 24456 leader_election.cc:302] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:26:59.301465 25454 raft_consensus.cc:2781] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.301584 25454 raft_consensus.cc:482] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.301682 25454 raft_consensus.cc:3037] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.301739 25401 raft_consensus.cc:504] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.302696 25401 leader_election.cc:288] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:26:59.303252 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8a25580bc74004b0e31c34f841d2cc" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:26:59.303459 24652 raft_consensus.cc:3037] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.304630 25454 raft_consensus.cc:504] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.305341 25454 leader_election.cc:288] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:26:59.305426 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8a25580bc74004b0e31c34f841d2cc" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:26:59.305825 24522 raft_consensus.cc:2454] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:26:59.305939 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5a87719c2544a5b0cb591ed9f46df8" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:26:59.306080 24652 raft_consensus.cc:2454] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:26:59.306156 24784 raft_consensus.cc:3037] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.306476 24324 leader_election.cc:302] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:26:59.306373 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8a25580bc74004b0e31c34f841d2cc" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:26:59.306639 24521 raft_consensus.cc:3037] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.306766 25401 raft_consensus.cc:2781] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.307026 25401 raft_consensus.cc:686] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:26:59.307296 25401 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:26:59.307814 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5a87719c2544a5b0cb591ed9f46df8" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:26:59.308075 24652 raft_consensus.cc:3037] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.309128 24784 raft_consensus.cc:2454] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:26:59.309590 24521 raft_consensus.cc:2454] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:26:59.309616 24456 leader_election.cc:302] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:26:59.309967 25454 raft_consensus.cc:2781] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.310127 25454 raft_consensus.cc:686] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:26:59.310436 25454 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:26:59.312013 24652 raft_consensus.cc:2454] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:26:59.313378 24242 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.315377 24239 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.423389 25413 raft_consensus.cc:482] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.423647 25413 raft_consensus.cc:504] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.425474 25413 leader_election.cc:288] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:26:59.433260 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a35079a106c4cca94cf65ffa93aefbf" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:26:59.433482 24652 raft_consensus.cc:2454] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:26:59.433984 24719 leader_election.cc:302] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 427345e9a65d417c84a773ac1925a18a; no voters:
I20240502 21:26:59.434048 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a35079a106c4cca94cf65ffa93aefbf" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:26:59.434252 24521 raft_consensus.cc:2454] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:26:59.434398 25413 raft_consensus.cc:2781] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.434528 25413 raft_consensus.cc:482] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.434644 25413 raft_consensus.cc:3037] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.436638 25413 raft_consensus.cc:504] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.437321 25413 leader_election.cc:288] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:26:59.437800 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a35079a106c4cca94cf65ffa93aefbf" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:26:59.437788 24652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a35079a106c4cca94cf65ffa93aefbf" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:26:59.437987 24521 raft_consensus.cc:3037] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.437995 24652 raft_consensus.cc:3037] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.439874 24521 raft_consensus.cc:2454] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:26:59.440083 24652 raft_consensus.cc:2454] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:26:59.440284 24717 leader_election.cc:302] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:26:59.440619 25413 raft_consensus.cc:2781] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.441005 25413 raft_consensus.cc:686] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Becoming Leader. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:26:59.441494 25413 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:26:59.445158 24239 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.799841 25454 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.807147 25454 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:59.845158 25501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:59.845546 25501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:59.853571 25501 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:59.874033 25401 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.884210 25519 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
Using table 'rebalance_test_table_1'
I20240502 21:26:59.972398 25413 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.024436 25548 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 21:27:00.095090 24719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.3:38427, user_credentials={real_user=slave}} blocked reactor thread for 47130.9us
INSERT report
rows total: 2000
time total: 292.646 ms
time per row: 0.146323 ms
I20240502 21:27:00.303867 24239 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:00.313482 24368 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.313747 24763 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.314028 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4aa176fd994cf0b55f304cf9d06fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.314216 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4aa176fd994cf0b55f304cf9d06fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.314198 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.314664 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4aa176fd994cf0b55f304cf9d06fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.315675 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 855b247f654d41c2bbb13abb25d35c1c (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.316063 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855b247f654d41c2bbb13abb25d35c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.319005 24762 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.319303 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.319882 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.320472 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 855b247f654d41c2bbb13abb25d35c1c (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.320927 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855b247f654d41c2bbb13abb25d35c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.325189 24369 tablet_service.cc:1467] Processing CreateTablet for tablet 855b247f654d41c2bbb13abb25d35c1c (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.325829 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855b247f654d41c2bbb13abb25d35c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.326884 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.327378 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.319500 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.331800 25576 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:00.334164 25576 tablet_bootstrap.cc:654] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.339407 25576 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:00.339567 25576 ts_tablet_manager.cc:1367] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:27:00.340370 25576 raft_consensus.cc:348] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.340584 25576 raft_consensus.cc:374] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.340662 25576 raft_consensus.cc:729] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.340895 25576 consensus_queue.cc:260] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.341463 25576 ts_tablet_manager.cc:1398] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:00.341691 25575 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:00.341895 25578 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:00.343640 25575 tablet_bootstrap.cc:654] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.343681 25578 tablet_bootstrap.cc:654] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.344991 25576 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:00.346720 25576 tablet_bootstrap.cc:654] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.352887 25578 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:00.353075 25578 ts_tablet_manager.cc:1367] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 21:27:00.353777 25578 raft_consensus.cc:348] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.353960 25578 raft_consensus.cc:374] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.354053 25578 raft_consensus.cc:729] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.354297 25578 consensus_queue.cc:260] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.354607 25581 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:00.354997 25578 ts_tablet_manager.cc:1398] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:00.356784 25581 tablet_bootstrap.cc:654] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.357077 25576 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:00.357300 25576 ts_tablet_manager.cc:1367] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240502 21:27:00.358289 25576 raft_consensus.cc:348] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.358636 25576 raft_consensus.cc:374] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.358738 25576 raft_consensus.cc:729] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.358915 25575 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:00.359156 25575 ts_tablet_manager.cc:1367] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240502 21:27:00.359110 25576 consensus_queue.cc:260] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.359941 25576 ts_tablet_manager.cc:1398] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:00.360301 25576 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:00.360327 25575 raft_consensus.cc:348] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.360664 25575 raft_consensus.cc:374] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.360806 25575 raft_consensus.cc:729] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.361117 25575 consensus_queue.cc:260] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.361812 25575 ts_tablet_manager.cc:1398] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:00.362399 25576 tablet_bootstrap.cc:654] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.362821 25575 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:00.364498 25575 tablet_bootstrap.cc:654] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.365402 25581 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:00.365617 25581 ts_tablet_manager.cc:1367] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240502 21:27:00.366643 25581 raft_consensus.cc:348] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.366930 25581 raft_consensus.cc:374] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.367044 25581 raft_consensus.cc:729] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.367367 25581 consensus_queue.cc:260] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.368245 25581 ts_tablet_manager.cc:1398] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:00.368626 25581 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:00.370545 25581 tablet_bootstrap.cc:654] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.370695 25576 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:00.370858 25576 ts_tablet_manager.cc:1367] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 21:27:00.371719 25576 raft_consensus.cc:348] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.371959 25576 raft_consensus.cc:374] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.372068 25576 raft_consensus.cc:729] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.372359 25576 consensus_queue.cc:260] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.372560 25581 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:00.372740 25581 ts_tablet_manager.cc:1367] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:00.373049 25576 ts_tablet_manager.cc:1398] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:00.373520 25581 raft_consensus.cc:348] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.373781 25581 raft_consensus.cc:374] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.373859 25581 raft_consensus.cc:729] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.374436 25575 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:00.374625 25575 ts_tablet_manager.cc:1367] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240502 21:27:00.374191 25581 consensus_queue.cc:260] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.375260 25581 ts_tablet_manager.cc:1398] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:00.375559 25581 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:00.375505 25575 raft_consensus.cc:348] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.375734 25575 raft_consensus.cc:374] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.375815 25575 raft_consensus.cc:729] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.376063 25575 consensus_queue.cc:260] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.376626 25575 ts_tablet_manager.cc:1398] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:00.377794 25581 tablet_bootstrap.cc:654] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.379846 25581 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:00.379977 25581 ts_tablet_manager.cc:1367] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 21:27:00.380645 25581 raft_consensus.cc:348] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.380803 25581 raft_consensus.cc:374] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.380903 25581 raft_consensus.cc:729] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.381091 25581 consensus_queue.cc:260] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.381484 25581 ts_tablet_manager.cc:1398] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:00.397717 25454 raft_consensus.cc:482] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.397938 25454 raft_consensus.cc:504] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.398582 25454 leader_election.cc:288] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:00.399283 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4aa176fd994cf0b55f304cf9d06fe0" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:00.399277 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4aa176fd994cf0b55f304cf9d06fe0" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:00.399540 24390 raft_consensus.cc:2454] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:00.399560 24784 raft_consensus.cc:2454] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:00.400036 24456 leader_election.cc:302] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:00.400322 25454 raft_consensus.cc:2781] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.400419 25454 raft_consensus.cc:482] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.400524 25454 raft_consensus.cc:3037] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.402667 25454 raft_consensus.cc:504] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.403247 25454 leader_election.cc:288] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:00.403767 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4aa176fd994cf0b55f304cf9d06fe0" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:00.403785 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4aa176fd994cf0b55f304cf9d06fe0" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:00.404000 24390 raft_consensus.cc:3037] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.404021 24784 raft_consensus.cc:3037] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.406937 24784 raft_consensus.cc:2454] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:00.406953 24390 raft_consensus.cc:2454] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:00.407488 24456 leader_election.cc:302] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:00.407801 25454 raft_consensus.cc:2781] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.407974 25454 raft_consensus.cc:686] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:00.408285 25454 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:00.410778 25413 raft_consensus.cc:482] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.410943 25413 raft_consensus.cc:504] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.411602 25413 leader_election.cc:288] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:00.411494 24239 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:00.412117 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:00.412333 24521 raft_consensus.cc:2454] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:00.413966 24717 leader_election.cc:302] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:00.414265 25413 raft_consensus.cc:2781] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.414426 25413 raft_consensus.cc:482] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.414528 25413 raft_consensus.cc:3037] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.416945 25413 raft_consensus.cc:504] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.417495 25413 leader_election.cc:288] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:00.417948 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:00.418123 24521 raft_consensus.cc:3037] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.420241 24521 raft_consensus.cc:2454] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:00.420681 24717 leader_election.cc:302] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:00.420873 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:00.421016 25413 raft_consensus.cc:2781] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.421159 24389 raft_consensus.cc:3037] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.421175 25413 raft_consensus.cc:686] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Becoming Leader. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:27:00.421469 25413 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:00.424422 24389 raft_consensus.cc:2454] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:00.424288 24239 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:00.424818 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:00.425055 24390 raft_consensus.cc:2362] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1. Re-sending same reply.
I20240502 21:27:00.425576 25519 raft_consensus.cc:482] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.425751 25519 raft_consensus.cc:504] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.426420 25519 leader_election.cc:288] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:00.427028 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:00.427307 24651 raft_consensus.cc:2454] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:00.427232 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:00.427515 24784 raft_consensus.cc:2454] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:00.427832 24324 leader_election.cc:302] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:00.428141 25519 raft_consensus.cc:2781] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.428277 25519 raft_consensus.cc:482] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.428372 25519 raft_consensus.cc:3037] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.430202 25519 raft_consensus.cc:504] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.431047 25519 leader_election.cc:288] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:00.431407 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:00.431581 24784 raft_consensus.cc:3037] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.432121 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:00.432339 24651 raft_consensus.cc:3037] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.434458 24784 raft_consensus.cc:2454] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:00.434897 24323 leader_election.cc:302] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:00.435067 24651 raft_consensus.cc:2454] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:00.435175 25519 raft_consensus.cc:2781] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.435377 25519 raft_consensus.cc:686] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:00.435710 25519 consensus_queue.cc:237] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:00.438776 24239 catalog_manager.cc:5484] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:00.789834 25454 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.805673 25600 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:00.826571 25592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:00.826951 25592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:00.834892 25592 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 21:27:00.882115 25519 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.887145 25519 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.986304 24389 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.986991 25413 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.987206 24521 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.994091 25413 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.114768 25595 mvcc.cc:205] Tried to move back new op lower bound from 7023350665153187840 to 7023350662848966656. Current Snapshot: MvccSnapshot[applied={T|T < 7023350665153187840}]
I20240502 21:27:01.147680 25639 mvcc.cc:205] Tried to move back new op lower bound from 7023350665730818048 to 7023350662848966656. Current Snapshot: MvccSnapshot[applied={T|T < 7023350665730818048}]
INSERT report
rows total: 2000
time total: 274.623 ms
time per row: 0.137312 ms
I20240502 21:27:01.267184 24240 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:01.277323 24369 tablet_service.cc:1467] Processing CreateTablet for tablet 8f912eb05c574368909e0b3a8edf058d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.277371 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 8f912eb05c574368909e0b3a8edf058d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.277454 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 8f912eb05c574368909e0b3a8edf058d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.277868 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f912eb05c574368909e0b3a8edf058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.277868 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f912eb05c574368909e0b3a8edf058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.277868 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f912eb05c574368909e0b3a8edf058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.279139 24631 tablet_service.cc:1467] Processing CreateTablet for tablet adde18faca794600bb4aa24f0cf744e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.279552 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde18faca794600bb4aa24f0cf744e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.282756 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.283154 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6e7eba855e48c0b0e18fa363ea01d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.285413 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.285975 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6e7eba855e48c0b0e18fa363ea01d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.292306 24762 tablet_service.cc:1467] Processing CreateTablet for tablet adde18faca794600bb4aa24f0cf744e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.292465 25665 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:01.292801 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde18faca794600bb4aa24f0cf744e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.293994 25665 tablet_bootstrap.cc:654] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.295945 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.296378 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6e7eba855e48c0b0e18fa363ea01d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.297778 25665 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:01.297665 24370 tablet_service.cc:1467] Processing CreateTablet for tablet adde18faca794600bb4aa24f0cf744e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.297942 25665 ts_tablet_manager.cc:1367] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:27:01.298125 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde18faca794600bb4aa24f0cf744e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.298811 25665 raft_consensus.cc:348] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.299041 25665 raft_consensus.cc:374] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.299134 25665 raft_consensus.cc:729] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.299381 25665 consensus_queue.cc:260] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.300055 25665 ts_tablet_manager.cc:1398] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:01.300338 25665 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:01.302183 25665 tablet_bootstrap.cc:654] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.306406 25669 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:01.307204 25668 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:01.309032 25668 tablet_bootstrap.cc:654] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.309929 25669 tablet_bootstrap.cc:654] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.314981 25666 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:01.315573 25669 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:01.315634 25665 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:01.315825 25669 ts_tablet_manager.cc:1367] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240502 21:27:01.315825 25665 ts_tablet_manager.cc:1367] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240502 21:27:01.316660 25669 raft_consensus.cc:348] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.316700 25665 raft_consensus.cc:348] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.316908 25665 raft_consensus.cc:374] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.316990 25665 raft_consensus.cc:729] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.317006 25669 raft_consensus.cc:374] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.317198 25669 raft_consensus.cc:729] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.317327 25665 consensus_queue.cc:260] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.317485 25669 consensus_queue.cc:260] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.318120 25665 ts_tablet_manager.cc:1398] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.318969 25669 ts_tablet_manager.cc:1398] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:01.319002 25666 tablet_bootstrap.cc:654] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.319339 25669 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:01.321027 25669 tablet_bootstrap.cc:654] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.322427 25665 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:01.322577 25668 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:01.322743 25668 ts_tablet_manager.cc:1367] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240502 21:27:01.323748 25668 raft_consensus.cc:348] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.323976 25668 raft_consensus.cc:374] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.324087 25668 raft_consensus.cc:729] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.324115 25665 tablet_bootstrap.cc:654] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.324466 25668 consensus_queue.cc:260] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.325263 25668 ts_tablet_manager.cc:1398] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.325585 25668 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:01.328651 25668 tablet_bootstrap.cc:654] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.329427 25665 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:01.329592 25665 ts_tablet_manager.cc:1367] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:27:01.329901 25666 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:01.330116 25666 ts_tablet_manager.cc:1367] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240502 21:27:01.330483 25665 raft_consensus.cc:348] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.330698 25665 raft_consensus.cc:374] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.330731 25669 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:01.330798 25665 raft_consensus.cc:729] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.331013 25669 ts_tablet_manager.cc:1367] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20240502 21:27:01.331043 25665 consensus_queue.cc:260] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.331754 25665 ts_tablet_manager.cc:1398] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:01.331969 25669 raft_consensus.cc:348] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.332157 25669 raft_consensus.cc:374] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.332235 25669 raft_consensus.cc:729] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.331225 25666 raft_consensus.cc:348] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.332362 25666 raft_consensus.cc:374] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.332482 25666 raft_consensus.cc:729] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.332475 25669 consensus_queue.cc:260] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.332785 25666 consensus_queue.cc:260] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.333125 25669 ts_tablet_manager.cc:1398] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:01.334211 25666 ts_tablet_manager.cc:1398] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:01.334520 25666 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:01.336196 25668 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:01.336434 25668 ts_tablet_manager.cc:1367] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 21:27:01.336861 25666 tablet_bootstrap.cc:654] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.338915 25666 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:01.338815 25668 raft_consensus.cc:348] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.339067 25666 ts_tablet_manager.cc:1367] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 21:27:01.339088 25668 raft_consensus.cc:374] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.339177 25668 raft_consensus.cc:729] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.339462 25668 consensus_queue.cc:260] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.339845 25666 raft_consensus.cc:348] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.340065 25666 raft_consensus.cc:374] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.340152 25666 raft_consensus.cc:729] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.340296 25668 ts_tablet_manager.cc:1398] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:01.340389 25666 consensus_queue.cc:260] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.341122 25666 ts_tablet_manager.cc:1398] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:01.443038 25621 raft_consensus.cc:482] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.443249 25621 raft_consensus.cc:504] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.444844 25621 leader_election.cc:288] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:01.448004 25462 raft_consensus.cc:482] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.448217 25462 raft_consensus.cc:504] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.449077 25462 leader_election.cc:288] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:01.449477 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f912eb05c574368909e0b3a8edf058d" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:01.449741 24389 raft_consensus.cc:2454] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:01.450326 24456 leader_election.cc:302] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:01.450541 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f912eb05c574368909e0b3a8edf058d" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:01.450662 25462 raft_consensus.cc:2781] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.450773 25462 raft_consensus.cc:482] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.450845 25462 raft_consensus.cc:3037] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.450816 24651 raft_consensus.cc:2454] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:01.453267 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f912eb05c574368909e0b3a8edf058d" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:01.453804 25462 raft_consensus.cc:504] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.453840 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f912eb05c574368909e0b3a8edf058d" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:01.454035 24389 raft_consensus.cc:2454] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:01.454195 24521 raft_consensus.cc:2379] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 427345e9a65d417c84a773ac1925a18a in current term 1: Already voted for candidate 7f8bb40905da4891b461220b95c66832 in this term.
I20240502 21:27:01.454538 24587 leader_election.cc:302] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:01.454780 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f912eb05c574368909e0b3a8edf058d" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:01.454983 24389 raft_consensus.cc:3037] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.455127 25462 leader_election.cc:288] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:01.455237 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f912eb05c574368909e0b3a8edf058d" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:01.455441 24651 raft_consensus.cc:3037] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.457902 24389 raft_consensus.cc:2454] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:01.458279 24651 raft_consensus.cc:2454] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:01.458431 24456 leader_election.cc:302] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:01.458474 25621 raft_consensus.cc:2741] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 21:27:01.458763 25462 raft_consensus.cc:2781] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.458891 25462 raft_consensus.cc:686] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:01.459180 25462 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.461951 24240 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.650271 25462 raft_consensus.cc:482] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.650502 25462 raft_consensus.cc:504] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.651175 25462 leader_election.cc:288] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:01.651698 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6e7eba855e48c0b0e18fa363ea01d4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:01.651731 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6e7eba855e48c0b0e18fa363ea01d4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:01.651911 24784 raft_consensus.cc:2454] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:01.651968 24651 raft_consensus.cc:2454] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:01.652398 24457 leader_election.cc:302] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:01.652676 25462 raft_consensus.cc:2781] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.652774 25462 raft_consensus.cc:482] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.652889 25462 raft_consensus.cc:3037] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.654796 25462 raft_consensus.cc:504] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.655417 25462 leader_election.cc:288] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:01.655771 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6e7eba855e48c0b0e18fa363ea01d4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:01.655900 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6e7eba855e48c0b0e18fa363ea01d4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:01.656002 24651 raft_consensus.cc:3037] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.656103 24784 raft_consensus.cc:3037] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.658265 24651 raft_consensus.cc:2454] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:01.658751 24457 leader_election.cc:302] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:01.658993 25462 raft_consensus.cc:2781] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.659149 25462 raft_consensus.cc:686] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:01.659276 24784 raft_consensus.cc:2454] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:01.659430 25462 consensus_queue.cc:237] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:01.662302 24240 catalog_manager.cc:5484] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:01.672784 25401 raft_consensus.cc:482] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.672966 25401 raft_consensus.cc:504] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.673646 25401 leader_election.cc:288] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:01.674142 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:01.674154 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:01.674401 24651 raft_consensus.cc:2454] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:01.674422 24784 raft_consensus.cc:2454] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:01.674846 24324 leader_election.cc:302] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:01.675144 25401 raft_consensus.cc:2781] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.675236 25401 raft_consensus.cc:482] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.675346 25401 raft_consensus.cc:3037] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.677183 25401 raft_consensus.cc:504] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.677732 25401 leader_election.cc:288] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:01.678196 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:01.678212 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:01.678413 24784 raft_consensus.cc:3037] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.678421 24651 raft_consensus.cc:3037] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.680441 24651 raft_consensus.cc:2454] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:01.680447 24784 raft_consensus.cc:2454] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:01.680859 24324 leader_election.cc:302] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:01.681129 25401 raft_consensus.cc:2781] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.681496 25401 raft_consensus.cc:686] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:01.681744 25401 consensus_queue.cc:237] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:01.684710 24240 catalog_manager.cc:5484] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.864126 25462 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.870494 25462 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:02.117889 25684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:02.118237 25684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:02.126096 25684 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 21:27:02.195276 25600 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.201792 25462 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.225858 25401 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.248296 25715 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:02.374040 24456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:33687, user_credentials={real_user=slave}} blocked reactor thread for 69137.5us
INSERT report
rows total: 2000
time total: 268.65 ms
time per row: 0.134325 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:03.485970 24245 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:03.496014 24369 tablet_service.cc:1467] Processing CreateTablet for tablet dc5c40e2bf0340958ecd3e7541209e95 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.496163 24501 tablet_service.cc:1467] Processing CreateTablet for tablet dc5c40e2bf0340958ecd3e7541209e95 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.496585 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5c40e2bf0340958ecd3e7541209e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.496631 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5c40e2bf0340958ecd3e7541209e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.498134 24633 tablet_service.cc:1467] Processing CreateTablet for tablet bdf23b7a397441b5947ce6babe3e8f62 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.498585 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.499261 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6a4a74e0544e0e99e734a145c91d3b (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.499430 24762 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6a4a74e0544e0e99e734a145c91d3b (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.499622 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6a4a74e0544e0e99e734a145c91d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.499848 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6a4a74e0544e0e99e734a145c91d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.502777 24764 tablet_service.cc:1467] Processing CreateTablet for tablet bdf23b7a397441b5947ce6babe3e8f62 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.503175 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.507336 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6a4a74e0544e0e99e734a145c91d3b (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.507656 24368 tablet_service.cc:1467] Processing CreateTablet for tablet bdf23b7a397441b5947ce6babe3e8f62 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.507792 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6a4a74e0544e0e99e734a145c91d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.508059 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.515535 25756 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:03.517933 25756 tablet_bootstrap.cc:654] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.518967 24631 tablet_service.cc:1467] Processing CreateTablet for tablet dc5c40e2bf0340958ecd3e7541209e95 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.519593 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5c40e2bf0340958ecd3e7541209e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.523206 25757 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:03.524044 25756 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:03.524288 25756 ts_tablet_manager.cc:1367] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 21:27:03.525283 25756 raft_consensus.cc:348] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.525558 25756 raft_consensus.cc:374] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.525701 25756 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.526260 25756 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.527738 25756 ts_tablet_manager.cc:1398] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:03.528038 25756 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:03.528965 25757 tablet_bootstrap.cc:654] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.529736 25756 tablet_bootstrap.cc:654] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.536726 25759 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:03.537338 25755 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:03.538615 25759 tablet_bootstrap.cc:654] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.539114 25756 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:03.539152 25755 tablet_bootstrap.cc:654] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.539382 25756 ts_tablet_manager.cc:1367] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.005s
I20240502 21:27:03.540472 25756 raft_consensus.cc:348] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.540760 25756 raft_consensus.cc:374] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.540884 25756 raft_consensus.cc:729] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.541218 25756 consensus_queue.cc:260] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.541836 25757 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:03.542101 25757 ts_tablet_manager.cc:1367] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240502 21:27:03.543067 25756 ts_tablet_manager.cc:1398] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:03.543534 25756 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:03.545635 25756 tablet_bootstrap.cc:654] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.545734 25759 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:03.545931 25759 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:27:03.546825 25755 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:03.546789 25759 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.547053 25759 raft_consensus.cc:374] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.547056 25755 ts_tablet_manager.cc:1367] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:27:03.547180 25759 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.547480 25759 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.548053 25755 raft_consensus.cc:348] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.548401 25755 raft_consensus.cc:374] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.548552 25755 raft_consensus.cc:729] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.548956 25755 consensus_queue.cc:260] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.549834 25755 ts_tablet_manager.cc:1398] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:03.549705 25757 raft_consensus.cc:348] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.549988 25757 raft_consensus.cc:374] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.550170 25757 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.550280 25755 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:03.552462 25727 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.552624 25727 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.552778 25757 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.553495 25727 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:03.554049 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:03.554260 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:03.554387 24784 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:03.555290 24323 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:03.555398 25756 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:03.555554 25756 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240502 21:27:03.555810 25755 tablet_bootstrap.cc:654] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
W20240502 21:27:03.555837 24322 leader_election.cc:341] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:03.556555 25727 raft_consensus.cc:2781] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.556483 25756 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.556701 25727 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.556797 25727 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.557051 25759 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20240502 21:27:03.558154 25759 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:03.559672 25727 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.560876 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:03.561120 24784 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.562165 25759 tablet_bootstrap.cc:654] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.563699 25757 ts_tablet_manager.cc:1398] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240502 21:27:03.563972 25757 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:03.564435 24784 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:03.565627 25757 tablet_bootstrap.cc:654] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.565598 24323 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:03.566555 25755 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:03.566735 25755 ts_tablet_manager.cc:1367] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 21:27:03.567533 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:03.567622 25755 raft_consensus.cc:348] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.567845 25755 raft_consensus.cc:374] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.567924 25755 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.568240 25755 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.568604 24521 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.569164 25727 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:03.569701 25727 raft_consensus.cc:2781] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.570186 25727 raft_consensus.cc:686] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:03.571102 25759 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:03.571300 25759 ts_tablet_manager.cc:1367] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240502 21:27:03.571390 24521 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:03.570804 25727 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:03.572544 25755 ts_tablet_manager.cc:1398] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:03.572772 25759 raft_consensus.cc:348] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.573014 25759 raft_consensus.cc:374] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.573109 25759 raft_consensus.cc:729] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.573391 25759 consensus_queue.cc:260] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.574023 25759 ts_tablet_manager.cc:1398] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:03.556708 25756 raft_consensus.cc:374] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.574298 25756 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.575829 25757 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:03.575716 25756 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.576004 25757 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 21:27:03.576290 25756 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 21:27:03.578074 25757 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.578730 24239 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.579296 25757 raft_consensus.cc:374] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.579418 25757 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.579756 25757 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.580394 25757 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:03.640291 25727 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.640529 25727 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.641090 25727 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:03.641646 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:03.641638 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:03.641904 24651 raft_consensus.cc:2454] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:03.642437 24324 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:03.641904 24784 raft_consensus.cc:2454] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:03.642765 25727 raft_consensus.cc:2781] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.642867 25727 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.642983 25727 raft_consensus.cc:3037] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.647240 25727 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.647897 25727 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:03.648188 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:03.648373 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:03.648414 24784 raft_consensus.cc:3037] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.648584 24651 raft_consensus.cc:3037] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.651407 24651 raft_consensus.cc:2454] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:03.651410 24784 raft_consensus.cc:2454] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:03.651836 24323 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:03.652186 25727 raft_consensus.cc:2781] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.652371 25727 raft_consensus.cc:686] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:03.652637 25727 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:03.655642 24245 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:03.683199 25765 raft_consensus.cc:482] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.683465 25765 raft_consensus.cc:504] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.684301 25765 leader_election.cc:288] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:03.684933 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:03.685025 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:03.685181 24389 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:03.685276 24521 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:03.685760 24586 leader_election.cc:302] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:03.686060 25765 raft_consensus.cc:2781] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.686205 25765 raft_consensus.cc:482] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.686311 25765 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.688479 25765 raft_consensus.cc:504] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.689152 25765 leader_election.cc:288] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:03.689653 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:03.689687 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:03.689854 24389 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.689896 24521 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.692054 24389 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:03.692566 24587 leader_election.cc:302] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:03.692889 25765 raft_consensus.cc:2781] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.692852 24521 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:03.693650 25765 raft_consensus.cc:686] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Leader. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:03.694130 25765 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:03.698287 24245 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a reported cstate change: term changed from 0 to 1, leader changed from <none> to 427345e9a65d417c84a773ac1925a18a (127.16.122.3). New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:03.964252 25727 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.968894 25776 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.046228 25781 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.050876 25768 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:04.069584 25774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:04.069901 25774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:04.078176 25774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:04.078164 25765 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.091228 25765 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 265.624 ms
time per row: 0.132812 ms
I20240502 21:27:04.501497 24246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:04.510834 24764 tablet_service.cc:1467] Processing CreateTablet for tablet d0ba1af1d3624d6387bf892873f3f67e (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.511070 24501 tablet_service.cc:1467] Processing CreateTablet for tablet d0ba1af1d3624d6387bf892873f3f67e (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.511348 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ba1af1d3624d6387bf892873f3f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.511528 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ba1af1d3624d6387bf892873f3f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.512684 24368 tablet_service.cc:1467] Processing CreateTablet for tablet 616e911db4384c17905713adc595ea0a (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.513150 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e911db4384c17905713adc595ea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.513041 24631 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.513482 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.514797 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 616e911db4384c17905713adc595ea0a (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.515192 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e911db4384c17905713adc595ea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.518868 24633 tablet_service.cc:1467] Processing CreateTablet for tablet d0ba1af1d3624d6387bf892873f3f67e (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.519115 24762 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.519362 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ba1af1d3624d6387bf892873f3f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.519629 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.523818 24500 tablet_service.cc:1467] Processing CreateTablet for tablet 616e911db4384c17905713adc595ea0a (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.524386 24502 tablet_service.cc:1467] Processing CreateTablet for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.524420 24500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e911db4384c17905713adc595ea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.525296 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.526332 25839 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:04.528159 25839 tablet_bootstrap.cc:654] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.547515 25842 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:04.552269 25842 tablet_bootstrap.cc:654] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.553718 25839 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:04.553933 25839 ts_tablet_manager.cc:1367] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20240502 21:27:04.553967 25838 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:04.554883 25839 raft_consensus.cc:348] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.555177 25839 raft_consensus.cc:374] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.555305 25839 raft_consensus.cc:729] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.555642 25839 consensus_queue.cc:260] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.555934 25838 tablet_bootstrap.cc:654] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.556456 25839 ts_tablet_manager.cc:1398] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:04.557228 25841 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:04.559378 25841 tablet_bootstrap.cc:654] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.559967 25842 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:04.560165 25842 ts_tablet_manager.cc:1367] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240502 21:27:04.561012 25842 raft_consensus.cc:348] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.561285 25842 raft_consensus.cc:374] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.561410 25842 raft_consensus.cc:729] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.561661 25842 consensus_queue.cc:260] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.562544 25842 ts_tablet_manager.cc:1398] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:04.562880 25842 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:04.565083 25842 tablet_bootstrap.cc:654] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.565920 25839 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:04.568156 25839 tablet_bootstrap.cc:654] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.574394 25839 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:04.574584 25839 ts_tablet_manager.cc:1367] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:27:04.575192 25842 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:04.575207 25841 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:04.575306 25838 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:04.575378 25841 ts_tablet_manager.cc:1367] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.011s
I20240502 21:27:04.575475 25838 ts_tablet_manager.cc:1367] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.005s
I20240502 21:27:04.575498 25842 ts_tablet_manager.cc:1367] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 21:27:04.575467 25839 raft_consensus.cc:348] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.575738 25839 raft_consensus.cc:374] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.575819 25839 raft_consensus.cc:729] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.576064 25839 consensus_queue.cc:260] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.576275 25842 raft_consensus.cc:348] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.576507 25842 raft_consensus.cc:374] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.576385 25838 raft_consensus.cc:348] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.576589 25842 raft_consensus.cc:729] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.576602 25838 raft_consensus.cc:374] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.576678 25839 ts_tablet_manager.cc:1398] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:04.576696 25838 raft_consensus.cc:729] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.576831 25841 raft_consensus.cc:348] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.576889 25842 consensus_queue.cc:260] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.577133 25841 raft_consensus.cc:374] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.576998 25838 consensus_queue.cc:260] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.577440 25842 ts_tablet_manager.cc:1398] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:04.577967 25842 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:04.578104 25841 raft_consensus.cc:729] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.578570 25838 ts_tablet_manager.cc:1398] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:04.578466 25841 consensus_queue.cc:260] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.579214 25841 ts_tablet_manager.cc:1398] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:04.579955 25842 tablet_bootstrap.cc:654] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.579953 25838 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:04.581719 25838 tablet_bootstrap.cc:654] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.582969 25842 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:04.583165 25842 ts_tablet_manager.cc:1367] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 21:27:04.583948 25842 raft_consensus.cc:348] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.584218 25842 raft_consensus.cc:374] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.584368 25842 raft_consensus.cc:729] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.584686 25842 consensus_queue.cc:260] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.585587 25842 ts_tablet_manager.cc:1398] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:04.586431 25838 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:04.586613 25838 ts_tablet_manager.cc:1367] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:27:04.587502 25838 raft_consensus.cc:348] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.587762 25838 raft_consensus.cc:374] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.587854 25838 raft_consensus.cc:729] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.588107 25838 consensus_queue.cc:260] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.589011 25838 ts_tablet_manager.cc:1398] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:04.589313 25838 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:04.590991 25838 tablet_bootstrap.cc:654] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.592813 25838 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:04.592962 25838 ts_tablet_manager.cc:1367] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:04.593575 25838 raft_consensus.cc:348] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.593750 25838 raft_consensus.cc:374] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.593847 25838 raft_consensus.cc:729] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.594061 25838 consensus_queue.cc:260] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.594614 25838 ts_tablet_manager.cc:1398] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:27:04.636610 25654 raft_consensus.cc:482] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.636806 25654 raft_consensus.cc:504] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.637450 25654 leader_election.cc:288] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:04.638017 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:04.638036 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:04.638264 24521 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:04.638273 24651 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:04.638759 24719 leader_election.cc:302] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 427345e9a65d417c84a773ac1925a18a; no voters:
I20240502 21:27:04.639048 25654 raft_consensus.cc:2781] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.639178 25654 raft_consensus.cc:482] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.639288 25654 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.641490 25654 raft_consensus.cc:504] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.642539 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:04.642547 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:04.642753 24651 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.642765 24521 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.644989 24651 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:04.645437 24719 leader_election.cc:302] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 427345e9a65d417c84a773ac1925a18a; no voters:
I20240502 21:27:04.645493 24521 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:04.642040 25654 leader_election.cc:288] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:04.646092 25654 raft_consensus.cc:2781] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.646224 25654 raft_consensus.cc:686] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Becoming Leader. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:27:04.646561 25654 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:04.650103 24245 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.683295 25768 raft_consensus.cc:482] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.683540 25768 raft_consensus.cc:504] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.684314 25768 leader_election.cc:288] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:04.684919 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e911db4384c17905713adc595ea0a" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:04.684926 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e911db4384c17905713adc595ea0a" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:04.685139 24521 raft_consensus.cc:2454] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:04.685207 24651 raft_consensus.cc:2454] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:04.685592 24322 leader_election.cc:302] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:04.685926 25768 raft_consensus.cc:2781] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.686018 25768 raft_consensus.cc:482] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.686132 25768 raft_consensus.cc:3037] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.688059 25768 raft_consensus.cc:504] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.688671 25768 leader_election.cc:288] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:04.689141 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e911db4384c17905713adc595ea0a" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:04.689249 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e911db4384c17905713adc595ea0a" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:04.689373 24651 raft_consensus.cc:3037] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.689479 24521 raft_consensus.cc:3037] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.692364 24651 raft_consensus.cc:2454] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:04.692374 24521 raft_consensus.cc:2454] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:04.692834 24324 leader_election.cc:302] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:04.693292 25776 raft_consensus.cc:2781] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.693465 25776 raft_consensus.cc:686] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:04.693786 25776 consensus_queue.cc:237] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:04.697101 24245 catalog_manager.cc:5484] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.721855 25850 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.722044 25850 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.722919 25850 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:04.723512 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:04.723518 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:04.723773 24783 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:04.723807 24651 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:04.724279 24456 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:04.724615 25850 raft_consensus.cc:2781] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.724746 25850 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.724857 25850 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.727181 25850 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.727800 25850 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:04.728261 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:04.728289 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:04.728467 24651 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.728506 24783 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.730482 24651 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:04.730981 24457 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:04.731434 24783 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:04.731616 25850 raft_consensus.cc:2781] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.732131 25850 raft_consensus.cc:686] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:04.732455 25850 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:04.735541 24246 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.034871 25852 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.042797 25852 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:05.109016 25856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:05.109362 25856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:05.117148 25856 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:05.139257 25776 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.145591 25781 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240502 21:27:05.174021 25850 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.180766 25735 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:05.401628 25880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.16.122.2:43159, user_credentials={real_user=slave}} blocked reactor thread for 47299.8us
INSERT report
rows total: 2000
time total: 280.086 ms
time per row: 0.140043 ms
I20240502 21:27:05.545809 24246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:05.554925 24368 tablet_service.cc:1467] Processing CreateTablet for tablet 53c1992a21224bac883da474f143c1d7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.555033 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 53c1992a21224bac883da474f143c1d7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.555464 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1992a21224bac883da474f143c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.555518 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1992a21224bac883da474f143c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.556371 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 049bec57d2474c328d2e26e16a4d33a9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.556818 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049bec57d2474c328d2e26e16a4d33a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.558712 24762 tablet_service.cc:1467] Processing CreateTablet for tablet 53c1992a21224bac883da474f143c1d7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.559113 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1992a21224bac883da474f143c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.559132 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 049bec57d2474c328d2e26e16a4d33a9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.559564 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049bec57d2474c328d2e26e16a4d33a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.559538 24763 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd91818e74e42acb276a3864753094d (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.559917 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd91818e74e42acb276a3864753094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.563375 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 049bec57d2474c328d2e26e16a4d33a9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.563795 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049bec57d2474c328d2e26e16a4d33a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.566411 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd91818e74e42acb276a3864753094d (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.566828 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd91818e74e42acb276a3864753094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.566741 24369 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd91818e74e42acb276a3864753094d (DEFAULT_TABLE table=rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.567090 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd91818e74e42acb276a3864753094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.570264 25920 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:05.572324 25920 tablet_bootstrap.cc:654] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.578528 25924 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:05.578720 25921 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:05.581247 25921 tablet_bootstrap.cc:654] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.585013 25924 tablet_bootstrap.cc:654] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.588363 25922 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:05.590938 25922 tablet_bootstrap.cc:654] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.592399 25924 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:05.592633 25924 ts_tablet_manager.cc:1367] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 21:27:05.593072 25920 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:05.593237 25920 ts_tablet_manager.cc:1367] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20240502 21:27:05.593722 25924 raft_consensus.cc:348] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.593936 25921 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:05.594095 25921 ts_tablet_manager.cc:1367] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240502 21:27:05.594143 25924 raft_consensus.cc:374] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.594201 25920 raft_consensus.cc:348] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.594323 25924 raft_consensus.cc:729] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.594506 25920 raft_consensus.cc:374] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.594664 25920 raft_consensus.cc:729] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.594833 25924 consensus_queue.cc:260] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.594964 25920 consensus_queue.cc:260] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.595623 25920 ts_tablet_manager.cc:1398] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:05.595788 25924 ts_tablet_manager.cc:1398] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:05.595937 25920 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:05.596184 25924 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:05.596347 25922 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:05.597604 25920 tablet_bootstrap.cc:654] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.597856 25922 ts_tablet_manager.cc:1367] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:27:05.598855 25922 raft_consensus.cc:348] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.599107 25922 raft_consensus.cc:374] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.599221 25922 raft_consensus.cc:729] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.599514 25922 consensus_queue.cc:260] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.600253 25922 ts_tablet_manager.cc:1398] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:05.600570 25922 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:05.602521 25922 tablet_bootstrap.cc:654] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.606941 25924 tablet_bootstrap.cc:654] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.595119 25921 raft_consensus.cc:348] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.608453 25921 raft_consensus.cc:374] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.608654 25921 raft_consensus.cc:729] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.609089 25921 consensus_queue.cc:260] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.610123 25921 ts_tablet_manager.cc:1398] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 21:27:05.613204 25924 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:05.616034 25924 ts_tablet_manager.cc:1367] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20240502 21:27:05.617216 25924 raft_consensus.cc:348] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.617548 25924 raft_consensus.cc:374] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.617640 25924 raft_consensus.cc:729] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.618050 25924 consensus_queue.cc:260] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.619035 25924 ts_tablet_manager.cc:1398] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:05.622256 25924 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:05.624832 25922 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:05.624883 25924 tablet_bootstrap.cc:654] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.625082 25920 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:05.625135 25922 ts_tablet_manager.cc:1367] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.007s
I20240502 21:27:05.625279 25920 ts_tablet_manager.cc:1367] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20240502 21:27:05.626158 25920 raft_consensus.cc:348] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.626436 25920 raft_consensus.cc:374] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.626521 25920 raft_consensus.cc:729] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.626279 25922 raft_consensus.cc:348] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.626771 25922 raft_consensus.cc:374] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.626899 25922 raft_consensus.cc:729] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.626833 25920 consensus_queue.cc:260] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.627179 25922 consensus_queue.cc:260] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.627523 25920 ts_tablet_manager.cc:1398] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:27:05.627746 25922 ts_tablet_manager.cc:1398] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:05.628041 25922 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:05.629292 25924 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:05.629452 25924 ts_tablet_manager.cc:1367] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 21:27:05.629689 25922 tablet_bootstrap.cc:654] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.630295 25924 raft_consensus.cc:348] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.630534 25924 raft_consensus.cc:374] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.630656 25924 raft_consensus.cc:729] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.631876 25924 consensus_queue.cc:260] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.632436 25924 ts_tablet_manager.cc:1398] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:05.633370 25922 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:05.633527 25922 ts_tablet_manager.cc:1367] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:05.634109 25922 raft_consensus.cc:348] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.634256 25922 raft_consensus.cc:374] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.634321 25922 raft_consensus.cc:729] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.634595 25922 consensus_queue.cc:260] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.635128 25922 ts_tablet_manager.cc:1398] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:05.655123 25768 raft_consensus.cc:482] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.655344 25768 raft_consensus.cc:504] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.656075 25768 leader_election.cc:288] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:05.656664 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd91818e74e42acb276a3864753094d" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:05.656744 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd91818e74e42acb276a3864753094d" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:05.656913 24521 raft_consensus.cc:2454] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:05.657016 24783 raft_consensus.cc:2454] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:05.657435 24322 leader_election.cc:302] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:05.657799 25768 raft_consensus.cc:2781] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.657891 25768 raft_consensus.cc:482] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.658002 25768 raft_consensus.cc:3037] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.660041 25768 raft_consensus.cc:504] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.660646 25768 leader_election.cc:288] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:05.661083 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd91818e74e42acb276a3864753094d" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:05.661141 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd91818e74e42acb276a3864753094d" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:05.661294 24521 raft_consensus.cc:3037] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.661340 24783 raft_consensus.cc:3037] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.663242 24521 raft_consensus.cc:2454] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:05.663254 24783 raft_consensus.cc:2454] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:05.663687 24322 leader_election.cc:302] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:05.664007 25768 raft_consensus.cc:2781] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.664161 25768 raft_consensus.cc:686] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:05.664434 25768 consensus_queue.cc:237] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.667568 24246 catalog_manager.cc:5484] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.721319 25735 raft_consensus.cc:482] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.721540 25735 raft_consensus.cc:504] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.722384 25735 leader_election.cc:288] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:05.722756 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1992a21224bac883da474f143c1d7" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:05.723049 24389 raft_consensus.cc:2454] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:05.723729 24456 leader_election.cc:302] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:05.724049 25735 raft_consensus.cc:2781] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.724155 25735 raft_consensus.cc:482] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.724084 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1992a21224bac883da474f143c1d7" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:05.724263 25735 raft_consensus.cc:3037] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.724335 24783 raft_consensus.cc:2454] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:05.727499 25735 raft_consensus.cc:504] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.728227 25735 leader_election.cc:288] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:05.728858 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1992a21224bac883da474f143c1d7" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:05.729127 24783 raft_consensus.cc:3037] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.729117 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1992a21224bac883da474f143c1d7" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:05.729368 24389 raft_consensus.cc:3037] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.731853 24389 raft_consensus.cc:2454] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:05.732240 24456 leader_election.cc:302] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:05.732548 25735 raft_consensus.cc:2781] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.732713 25735 raft_consensus.cc:686] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:05.732738 24783 raft_consensus.cc:2454] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:05.732995 25735 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:05.736023 24246 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.774291 25765 raft_consensus.cc:482] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.774550 25765 raft_consensus.cc:504] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.775226 25765 leader_election.cc:288] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:05.776494 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049bec57d2474c328d2e26e16a4d33a9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:05.776777 24389 raft_consensus.cc:2454] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:05.777218 24587 leader_election.cc:302] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:05.777580 25765 raft_consensus.cc:2781] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.777680 25765 raft_consensus.cc:482] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.777804 25765 raft_consensus.cc:3037] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.780079 25765 raft_consensus.cc:504] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.780762 25765 leader_election.cc:288] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:05.781312 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049bec57d2474c328d2e26e16a4d33a9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:05.781538 24389 raft_consensus.cc:3037] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.781965 24784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049bec57d2474c328d2e26e16a4d33a9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:05.781947 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049bec57d2474c328d2e26e16a4d33a9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:05.782208 24784 raft_consensus.cc:3037] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.783890 24389 raft_consensus.cc:2454] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:05.784126 24784 raft_consensus.cc:2454] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:05.784309 24587 leader_election.cc:302] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:05.784722 25765 raft_consensus.cc:2781] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.785132 25765 raft_consensus.cc:686] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Leader. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:05.785483 25765 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:05.788352 24246 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: term changed from 0 to 1, leader changed from <none> to 427345e9a65d417c84a773ac1925a18a (127.16.122.3). New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:06.112644 25768 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.118482 25781 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:06.172940 25938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:06.173302 25938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:06.181512 25938 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:06.191936 25765 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.200819 25765 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.208112 25850 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.233913 25850 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 251.832 ms
time per row: 0.125916 ms
I20240502 21:27:06.633901 24244 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:06.644209 24369 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee3718b120e4fad9ae03e47d3f1fffa (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.644754 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee3718b120e4fad9ae03e47d3f1fffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.645241 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 82fb425a5430426f8030ba5497736b11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.645697 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb425a5430426f8030ba5497736b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.646093 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee3718b120e4fad9ae03e47d3f1fffa (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.646576 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee3718b120e4fad9ae03e47d3f1fffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.647433 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 477d9df7d3a74b8584f3669be3256e00 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.647820 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.650916 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 477d9df7d3a74b8584f3669be3256e00 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.651387 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.655465 24631 tablet_service.cc:1467] Processing CreateTablet for tablet 477d9df7d3a74b8584f3669be3256e00 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.655896 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.658731 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 82fb425a5430426f8030ba5497736b11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.659102 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb425a5430426f8030ba5497736b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.645668 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee3718b120e4fad9ae03e47d3f1fffa (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.663195 24762 tablet_service.cc:1467] Processing CreateTablet for tablet 82fb425a5430426f8030ba5497736b11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.663507 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee3718b120e4fad9ae03e47d3f1fffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.663615 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb425a5430426f8030ba5497736b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.674928 26005 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:06.677311 26005 tablet_bootstrap.cc:654] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.681913 26006 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:06.683686 26006 tablet_bootstrap.cc:654] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.689288 26004 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:06.691309 26004 tablet_bootstrap.cc:654] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.700377 26004 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:06.700603 26004 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 21:27:06.701598 26005 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:06.701479 26004 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.701709 26004 raft_consensus.cc:374] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.701771 26005 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.009s
I20240502 21:27:06.701807 26004 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.702078 26004 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.702651 26005 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.702854 26004 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:06.702880 26005 raft_consensus.cc:374] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.702968 26005 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.703310 26004 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:06.704814 26004 tablet_bootstrap.cc:654] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.710907 26003 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:06.711213 26005 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.711917 26005 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:27:06.712312 26004 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:06.712538 26004 ts_tablet_manager.cc:1367] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.008s
I20240502 21:27:06.712795 26005 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:06.712937 26006 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:06.713076 26003 tablet_bootstrap.cc:654] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.713096 26006 ts_tablet_manager.cc:1367] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240502 21:27:06.713948 26006 raft_consensus.cc:348] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.714186 26006 raft_consensus.cc:374] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.714298 26006 raft_consensus.cc:729] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.714705 26005 tablet_bootstrap.cc:654] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.714591 26006 consensus_queue.cc:260] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.715206 26006 ts_tablet_manager.cc:1398] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:06.715524 26006 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:06.717352 26006 tablet_bootstrap.cc:654] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.718600 26004 raft_consensus.cc:348] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.718914 26004 raft_consensus.cc:374] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.719044 26004 raft_consensus.cc:729] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.719457 26004 consensus_queue.cc:260] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.721293 26004 ts_tablet_manager.cc:1398] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:06.723763 26006 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:06.723985 26006 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 21:27:06.725019 26006 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.725332 26006 raft_consensus.cc:374] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.725473 26006 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.725554 26003 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:06.725760 26003 ts_tablet_manager.cc:1367] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20240502 21:27:06.725853 26006 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.726481 26006 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:06.730281 26003 raft_consensus.cc:348] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.730623 26003 raft_consensus.cc:374] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.730742 26003 raft_consensus.cc:729] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.731006 26003 consensus_queue.cc:260] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.731642 26003 ts_tablet_manager.cc:1398] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:06.731971 26003 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:06.733371 26003 tablet_bootstrap.cc:654] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.739465 26003 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:06.739641 26003 ts_tablet_manager.cc:1367] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 21:27:06.740301 26003 raft_consensus.cc:348] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.740528 26003 raft_consensus.cc:374] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.740619 26003 raft_consensus.cc:729] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.741287 26005 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:06.741464 26005 ts_tablet_manager.cc:1367] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20240502 21:27:06.742103 26005 raft_consensus.cc:348] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.742288 26005 raft_consensus.cc:374] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.742405 26005 raft_consensus.cc:729] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.742511 26003 consensus_queue.cc:260] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.742660 26005 consensus_queue.cc:260] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.743103 26003 ts_tablet_manager.cc:1398] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:06.744321 26005 ts_tablet_manager.cc:1398] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:06.744585 26005 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:06.746852 26005 tablet_bootstrap.cc:654] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.749372 26005 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:06.749524 26005 ts_tablet_manager.cc:1367] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:06.750145 25992 raft_consensus.cc:482] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.750324 25992 raft_consensus.cc:504] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.750453 26005 raft_consensus.cc:348] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.750655 26005 raft_consensus.cc:374] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.750748 26005 raft_consensus.cc:729] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.750983 26005 consensus_queue.cc:260] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.751144 25992 leader_election.cc:288] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:06.751513 26005 ts_tablet_manager.cc:1398] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:06.751936 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477d9df7d3a74b8584f3669be3256e00" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:06.752151 24783 raft_consensus.cc:2454] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:06.752200 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477d9df7d3a74b8584f3669be3256e00" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:06.752411 24651 raft_consensus.cc:2454] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:06.752624 24323 leader_election.cc:302] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:06.752930 25992 raft_consensus.cc:2781] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.753067 25992 raft_consensus.cc:482] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.753150 25992 raft_consensus.cc:3037] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.756857 25992 raft_consensus.cc:504] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.757467 25992 leader_election.cc:288] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:06.758024 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477d9df7d3a74b8584f3669be3256e00" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:06.758044 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477d9df7d3a74b8584f3669be3256e00" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:06.758211 24651 raft_consensus.cc:3037] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.758262 24783 raft_consensus.cc:3037] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.760603 24651 raft_consensus.cc:2454] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:06.760990 24324 leader_election.cc:302] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:06.761090 24783 raft_consensus.cc:2454] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:06.761270 25992 raft_consensus.cc:2781] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.761418 25992 raft_consensus.cc:686] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:06.761742 25992 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:06.765100 24244 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:06.801708 25850 raft_consensus.cc:482] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.801911 25850 raft_consensus.cc:504] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.802628 25850 leader_election.cc:288] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:06.803216 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee3718b120e4fad9ae03e47d3f1fffa" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:06.803252 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee3718b120e4fad9ae03e47d3f1fffa" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:06.803504 24388 raft_consensus.cc:2454] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:06.803524 24651 raft_consensus.cc:2454] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:06.804013 24457 leader_election.cc:302] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:06.804311 25850 raft_consensus.cc:2781] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.804432 25850 raft_consensus.cc:482] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.804530 25850 raft_consensus.cc:3037] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.804828 25936 raft_consensus.cc:482] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.805004 25936 raft_consensus.cc:504] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.805759 25936 leader_election.cc:288] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:06.806594 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb425a5430426f8030ba5497736b11" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:06.806815 24521 raft_consensus.cc:2454] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:06.807173 25850 raft_consensus.cc:504] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.807296 24586 leader_election.cc:302] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:06.807277 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb425a5430426f8030ba5497736b11" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:06.807541 24783 raft_consensus.cc:2454] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:06.807636 25936 raft_consensus.cc:2781] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.807762 25936 raft_consensus.cc:482] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.807862 25936 raft_consensus.cc:3037] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.808444 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee3718b120e4fad9ae03e47d3f1fffa" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:06.808514 25850 leader_election.cc:288] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:06.808669 24651 raft_consensus.cc:3037] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.809136 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee3718b120e4fad9ae03e47d3f1fffa" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:06.809352 24388 raft_consensus.cc:3037] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.810307 25936 raft_consensus.cc:504] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.810969 25936 leader_election.cc:288] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:06.811573 24651 raft_consensus.cc:2454] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:06.811962 24457 leader_election.cc:302] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:06.812181 24388 raft_consensus.cc:2454] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:06.812702 25850 raft_consensus.cc:2781] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.812963 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb425a5430426f8030ba5497736b11" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:06.813162 24783 raft_consensus.cc:3037] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.812877 25850 raft_consensus.cc:686] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:06.813150 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb425a5430426f8030ba5497736b11" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:06.813354 24521 raft_consensus.cc:3037] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.813498 25850 consensus_queue.cc:237] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:06.815645 24783 raft_consensus.cc:2454] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:06.816094 24587 leader_election.cc:302] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 427345e9a65d417c84a773ac1925a18a; no voters:
I20240502 21:27:06.816299 24521 raft_consensus.cc:2454] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:06.816501 25936 raft_consensus.cc:2781] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.816679 25936 raft_consensus.cc:686] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Leader. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:06.817006 25936 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } }
I20240502 21:27:06.817045 24244 catalog_manager.cc:5484] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:06.820359 24240 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: term changed from 0 to 1, leader changed from <none> to 427345e9a65d417c84a773ac1925a18a (127.16.122.3). New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:07.192328 26019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:07.192704 26019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:07.203714 26019 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:07.209766 25995 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.234313 25995 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 21:27:07.285055 25936 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.291805 25765 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.391191 24388 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:07.391750 24652 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:07.392622 25850 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.393013 25735 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:07.490780 24323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:33687, user_credentials={real_user=slave}} blocked reactor thread for 85690.8us
I20240502 21:27:07.495076 26048 mvcc.cc:205] Tried to move back new op lower bound from 7023350691387084800 to 7023350689030754304. Current Snapshot: MvccSnapshot[applied={T|T < 7023350691387084800}]
INSERT report
rows total: 2000
time total: 267.996 ms
time per row: 0.133998 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:08.631269 24245 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:08.641117 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.641654 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf38bbaaaa340f89d766a843c6bcaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.641688 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.642134 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf38bbaaaa340f89d766a843c6bcaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.642066 24501 tablet_service.cc:1467] Processing CreateTablet for tablet ff7e019fa82d4648afaf130d3f6343d6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.642419 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e019fa82d4648afaf130d3f6343d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.643060 24762 tablet_service.cc:1467] Processing CreateTablet for tablet ff7e019fa82d4648afaf130d3f6343d6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.643481 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e019fa82d4648afaf130d3f6343d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.645030 24764 tablet_service.cc:1467] Processing CreateTablet for tablet 566929e603d54be4bf931413f0fa1d7d (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.645076 24631 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.645501 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf38bbaaaa340f89d766a843c6bcaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.646061 24368 tablet_service.cc:1467] Processing CreateTablet for tablet 566929e603d54be4bf931413f0fa1d7d (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.646065 24369 tablet_service.cc:1467] Processing CreateTablet for tablet ff7e019fa82d4648afaf130d3f6343d6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.646554 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.649919 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 566929e603d54be4bf931413f0fa1d7d (DEFAULT_TABLE table=rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:08.650277 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.650576 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.650763 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e019fa82d4648afaf130d3f6343d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.658691 26091 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:08.664947 26091 tablet_bootstrap.cc:654] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.672328 26094 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:08.672691 26095 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:08.674093 26095 tablet_bootstrap.cc:654] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.675516 26092 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:08.675558 26094 tablet_bootstrap.cc:654] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.675916 26095 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:08.676065 26095 ts_tablet_manager.cc:1367] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:27:08.676785 26095 raft_consensus.cc:348] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.676966 26095 raft_consensus.cc:374] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.677078 26095 raft_consensus.cc:729] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.677282 26092 tablet_bootstrap.cc:654] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.677346 26095 consensus_queue.cc:260] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.677456 26091 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:08.677637 26091 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240502 21:27:08.677976 26095 ts_tablet_manager.cc:1398] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:08.678421 26095 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:08.678483 26091 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.678777 26091 raft_consensus.cc:374] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.678910 26091 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.679224 26091 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.679997 26091 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:27:08.680286 26095 tablet_bootstrap.cc:654] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.680336 26091 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:08.682276 26091 tablet_bootstrap.cc:654] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.688256 26095 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:08.688491 26095 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 21:27:08.689378 26095 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.689651 26095 raft_consensus.cc:374] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.689757 26095 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.690034 26095 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.690680 26095 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:08.693969 26092 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:08.694221 26092 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20240502 21:27:08.695154 26092 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.695474 26092 raft_consensus.cc:374] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.695573 26092 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.695802 26092 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.696328 26092 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:08.696749 26092 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:08.698662 26092 tablet_bootstrap.cc:654] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.699256 26094 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:08.699501 26094 ts_tablet_manager.cc:1367] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240502 21:27:08.700524 26094 raft_consensus.cc:348] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.700901 26094 raft_consensus.cc:374] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.701084 26094 raft_consensus.cc:729] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.701457 26094 consensus_queue.cc:260] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.701881 26091 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:08.702100 26091 ts_tablet_manager.cc:1367] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.005s
I20240502 21:27:08.702394 26094 ts_tablet_manager.cc:1398] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:08.703037 26091 raft_consensus.cc:348] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.703275 26091 raft_consensus.cc:374] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.703358 26091 raft_consensus.cc:729] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.703807 26091 consensus_queue.cc:260] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.703990 26094 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:08.704459 26091 ts_tablet_manager.cc:1398] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:08.704952 26091 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:08.706713 26091 tablet_bootstrap.cc:654] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.708036 26094 tablet_bootstrap.cc:654] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.712689 26092 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:08.712894 26092 ts_tablet_manager.cc:1367] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240502 21:27:08.713773 26092 raft_consensus.cc:348] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.714064 26092 raft_consensus.cc:374] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.714156 26092 raft_consensus.cc:729] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.714442 26092 consensus_queue.cc:260] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.715152 26092 ts_tablet_manager.cc:1398] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:08.716449 26091 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:08.716591 26091 ts_tablet_manager.cc:1367] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240502 21:27:08.717398 26091 raft_consensus.cc:348] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.717674 26091 raft_consensus.cc:374] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.717820 26091 raft_consensus.cc:729] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.718144 26091 consensus_queue.cc:260] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.719358 26091 ts_tablet_manager.cc:1398] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:08.719626 26094 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:08.719784 26094 ts_tablet_manager.cc:1367] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 21:27:08.720405 26094 raft_consensus.cc:348] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.720613 26094 raft_consensus.cc:374] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.720693 26094 raft_consensus.cc:729] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.720937 26094 consensus_queue.cc:260] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.721463 26094 ts_tablet_manager.cc:1398] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:08.765708 26068 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.765890 26068 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.766629 26068 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:08.767069 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:08.767153 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:08.767328 24388 raft_consensus.cc:2454] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:08.767352 24783 raft_consensus.cc:2454] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:08.767761 24587 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 427345e9a65d417c84a773ac1925a18a; no voters:
I20240502 21:27:08.768045 26068 raft_consensus.cc:2781] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.768155 26068 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.768246 26068 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.770479 26068 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.771025 26068 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:08.771440 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:08.771672 24388 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.771636 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:08.771862 24783 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.774646 24388 raft_consensus.cc:2454] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:08.775076 24587 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:08.775193 24783 raft_consensus.cc:2454] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:08.775436 26068 raft_consensus.cc:2781] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.775846 26068 raft_consensus.cc:686] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Leader. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:08.776173 26068 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:08.779153 24244 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: term changed from 0 to 1, leader changed from <none> to 427345e9a65d417c84a773ac1925a18a (127.16.122.3). New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:08.974313 25912 raft_consensus.cc:482] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.974563 25912 raft_consensus.cc:504] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.975245 25912 leader_election.cc:288] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:08.975883 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e019fa82d4648afaf130d3f6343d6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:08.975883 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e019fa82d4648afaf130d3f6343d6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:08.976162 24389 raft_consensus.cc:2454] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:08.976162 24521 raft_consensus.cc:2454] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:08.976655 24718 leader_election.cc:302] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:08.976954 25912 raft_consensus.cc:2781] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.977080 25912 raft_consensus.cc:482] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.977182 25912 raft_consensus.cc:3037] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.979686 25912 raft_consensus.cc:504] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.980314 25912 leader_election.cc:288] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:08.980778 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e019fa82d4648afaf130d3f6343d6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:08.980863 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7e019fa82d4648afaf130d3f6343d6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:08.981034 24521 raft_consensus.cc:3037] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.981096 24389 raft_consensus.cc:3037] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.983953 24521 raft_consensus.cc:2454] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:08.983953 24389 raft_consensus.cc:2454] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:08.984424 24717 leader_election.cc:302] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:08.984777 25912 raft_consensus.cc:2781] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.985163 25912 raft_consensus.cc:686] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Becoming Leader. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:27:08.985494 25912 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:08.988590 24244 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:09.064738 26068 raft_consensus.cc:482] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.064992 26068 raft_consensus.cc:504] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:09.065694 26068 leader_election.cc:288] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:09.066239 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf38bbaaaa340f89d766a843c6bcaf9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:09.066293 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf38bbaaaa340f89d766a843c6bcaf9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:09.066545 24389 raft_consensus.cc:2454] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:09.066543 24522 raft_consensus.cc:2454] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 0.
I20240502 21:27:09.066927 24587 leader_election.cc:302] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:09.067209 26068 raft_consensus.cc:2781] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.067299 26068 raft_consensus.cc:482] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.067420 26068 raft_consensus.cc:3037] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.069301 26068 raft_consensus.cc:504] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:09.069913 26068 leader_election.cc:288] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:09.070473 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf38bbaaaa340f89d766a843c6bcaf9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:09.070523 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf38bbaaaa340f89d766a843c6bcaf9" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:09.070700 24389 raft_consensus.cc:3037] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.070729 24522 raft_consensus.cc:3037] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.072911 24389 raft_consensus.cc:2454] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:09.072935 24522 raft_consensus.cc:2454] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427345e9a65d417c84a773ac1925a18a in term 1.
I20240502 21:27:09.073323 24586 leader_election.cc:302] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:09.073678 26068 raft_consensus.cc:2781] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.073864 26068 raft_consensus.cc:686] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Leader. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:09.074153 26068 consensus_queue.cc:237] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:09.077418 24244 catalog_manager.cc:5484] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: term changed from 0 to 1, leader changed from <none> to 427345e9a65d417c84a773ac1925a18a (127.16.122.3). New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:09.173862 26068 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.183833 26068 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.480075 25912 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.484864 25912 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:09.500831 26110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:09.501168 26110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:09.508783 26110 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:09.539198 26068 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 21:27:09.544575 26105 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 216.403 ms
time per row: 0.108201 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:10.849056 24247 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58732:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:10.859071 24764 tablet_service.cc:1467] Processing CreateTablet for tablet e847a796029d4edea79694f5402a291a (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.859607 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e847a796029d4edea79694f5402a291a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.859666 24631 tablet_service.cc:1467] Processing CreateTablet for tablet e847a796029d4edea79694f5402a291a (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.860074 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e847a796029d4edea79694f5402a291a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.860975 24501 tablet_service.cc:1467] Processing CreateTablet for tablet e847a796029d4edea79694f5402a291a (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.861128 24500 tablet_service.cc:1467] Processing CreateTablet for tablet 98b5668e0d504db5a7874f041c14dbfa (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.861397 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e847a796029d4edea79694f5402a291a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.862254 24633 tablet_service.cc:1467] Processing CreateTablet for tablet 9963dcea478749e0a6e076fac6537fc4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.862545 24500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.862668 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.863050 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 9963dcea478749e0a6e076fac6537fc4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.863426 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.866799 24370 tablet_service.cc:1467] Processing CreateTablet for tablet 98b5668e0d504db5a7874f041c14dbfa (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.867223 24370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.867193 24762 tablet_service.cc:1467] Processing CreateTablet for tablet 9963dcea478749e0a6e076fac6537fc4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.867599 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.862301 24632 tablet_service.cc:1467] Processing CreateTablet for tablet 98b5668e0d504db5a7874f041c14dbfa (DEFAULT_TABLE table=rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.872890 26183 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:10.875907 26183 tablet_bootstrap.cc:654] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.875983 26182 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:10.876677 24632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.878137 26182 tablet_bootstrap.cc:654] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.893028 26186 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:10.896775 26186 tablet_bootstrap.cc:654] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.898870 26187 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
I20240502 21:27:10.900070 26182 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:10.900270 26182 ts_tablet_manager.cc:1367] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.009s
I20240502 21:27:10.900894 26182 raft_consensus.cc:348] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.901027 26183 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:10.901090 26182 raft_consensus.cc:374] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.901188 26182 raft_consensus.cc:729] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.901232 26183 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.008s
I20240502 21:27:10.901449 26182 consensus_queue.cc:260] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.902305 26183 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:10.902725 26183 raft_consensus.cc:374] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.902923 26183 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.903277 26183 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:10.904121 26183 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:10.904557 26183 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:10.904829 26182 ts_tablet_manager.cc:1398] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:10.907649 26182 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:10.908321 26183 tablet_bootstrap.cc:654] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.909428 26182 tablet_bootstrap.cc:654] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.911474 26187 tablet_bootstrap.cc:654] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.911478 26183 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: No bootstrap required, opened a new log
I20240502 21:27:10.911512 26186 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:10.911651 26183 ts_tablet_manager.cc:1367] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 21:27:10.911653 26186 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.011s
I20240502 21:27:10.912509 26186 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:10.912406 26183 raft_consensus.cc:348] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.912722 26186 raft_consensus.cc:374] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.912741 26183 raft_consensus.cc:374] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.912814 26183 raft_consensus.cc:729] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.912824 26186 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.913097 26183 consensus_queue.cc:260] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.913435 26187 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: No bootstrap required, opened a new log
I20240502 21:27:10.913585 26187 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20240502 21:27:10.913698 26186 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:10.913894 26183 ts_tablet_manager.cc:1398] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:10.914374 26187 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:10.914598 26187 raft_consensus.cc:374] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.914696 26187 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.914908 26187 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:10.915093 26186 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:10.915414 26186 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:10.915652 26187 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:10.917966 26182 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:10.917979 26186 tablet_bootstrap.cc:654] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.918195 26182 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240502 21:27:10.919109 26182 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:10.919427 26182 raft_consensus.cc:374] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.919550 26182 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.919878 26182 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:10.922973 26182 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:10.923488 26182 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:10.923748 26186 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:10.923920 26186 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240502 21:27:10.924708 26186 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:10.924952 26186 raft_consensus.cc:374] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.925048 26186 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.925303 26186 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:10.925846 26182 tablet_bootstrap.cc:654] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.926041 26186 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:10.926385 26186 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:10.927763 26186 tablet_bootstrap.cc:654] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.929636 26182 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: No bootstrap required, opened a new log
I20240502 21:27:10.929811 26182 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 21:27:10.930802 26182 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:10.930958 26182 raft_consensus.cc:374] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.931049 26182 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.931278 26182 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:10.931809 26182 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:10.935436 26186 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: No bootstrap required, opened a new log
I20240502 21:27:10.935591 26186 ts_tablet_manager.cc:1367] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 21:27:10.936267 26186 raft_consensus.cc:348] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.936475 26186 raft_consensus.cc:374] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.936578 26186 raft_consensus.cc:729] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.936796 26186 consensus_queue.cc:260] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.937300 26186 ts_tablet_manager.cc:1398] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:10.973806 26107 raft_consensus.cc:482] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.974030 26107 raft_consensus.cc:504] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.974877 26107 leader_election.cc:288] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:10.975636 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e847a796029d4edea79694f5402a291a" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:10.975636 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e847a796029d4edea79694f5402a291a" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:10.975932 24522 raft_consensus.cc:2454] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:10.975932 24653 raft_consensus.cc:2454] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 0.
I20240502 21:27:10.976524 24717 leader_election.cc:302] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:10.976852 26107 raft_consensus.cc:2781] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.976971 26107 raft_consensus.cc:482] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.977069 26107 raft_consensus.cc:3037] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.979503 26107 raft_consensus.cc:504] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.980129 26107 leader_election.cc:288] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:10.980680 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e847a796029d4edea79694f5402a291a" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:10.980715 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e847a796029d4edea79694f5402a291a" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:10.980907 24522 raft_consensus.cc:3037] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.980931 24653 raft_consensus.cc:3037] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.982843 24522 raft_consensus.cc:2454] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:10.982935 24653 raft_consensus.cc:2454] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 1.
I20240502 21:27:10.983319 24717 leader_election.cc:302] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:10.983697 26107 raft_consensus.cc:2781] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.984102 26107 raft_consensus.cc:686] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Becoming Leader. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:27:10.984447 26107 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }
I20240502 21:27:10.987987 24240 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4). New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:11.040438 26086 raft_consensus.cc:482] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.040607 26086 raft_consensus.cc:504] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:11.041221 26086 leader_election.cc:288] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:11.041795 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b5668e0d504db5a7874f041c14dbfa" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:11.041832 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b5668e0d504db5a7874f041c14dbfa" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:11.041991 24522 raft_consensus.cc:2454] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:11.042121 24653 raft_consensus.cc:2454] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 0.
I20240502 21:27:11.042474 24322 leader_election.cc:302] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:11.042774 26086 raft_consensus.cc:2781] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.042891 26086 raft_consensus.cc:482] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.042994 26086 raft_consensus.cc:3037] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.045410 26086 raft_consensus.cc:504] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:11.045993 26086 leader_election.cc:288] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:11.046401 24522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b5668e0d504db5a7874f041c14dbfa" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:11.046509 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b5668e0d504db5a7874f041c14dbfa" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:11.046603 24522 raft_consensus.cc:3037] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.046715 24653 raft_consensus.cc:3037] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.048664 24522 raft_consensus.cc:2454] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:11.048789 24653 raft_consensus.cc:2454] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e856847c3f494e6aa57f9cecf117ce4d in term 1.
I20240502 21:27:11.049079 24322 leader_election.cc:302] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:11.049762 26197 raft_consensus.cc:2781] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.049918 26197 raft_consensus.cc:686] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Leader. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:11.050236 26197 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } }
I20240502 21:27:11.053661 24240 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: term changed from 0 to 1, leader changed from <none> to e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1). New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:11.107812 26181 raft_consensus.cc:482] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.108080 26181 raft_consensus.cc:504] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:11.108814 26181 leader_election.cc:288] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:11.109452 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9963dcea478749e0a6e076fac6537fc4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:11.109501 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9963dcea478749e0a6e076fac6537fc4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:11.109719 24783 raft_consensus.cc:2454] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:11.109767 24653 raft_consensus.cc:2454] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 0.
I20240502 21:27:11.110206 24457 leader_election.cc:302] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:11.110520 26181 raft_consensus.cc:2781] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.110633 26181 raft_consensus.cc:482] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.110702 26181 raft_consensus.cc:3037] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.112665 26181 raft_consensus.cc:504] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:11.113260 26181 leader_election.cc:288] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:11.113734 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9963dcea478749e0a6e076fac6537fc4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:11.113816 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9963dcea478749e0a6e076fac6537fc4" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:11.113955 24653 raft_consensus.cc:3037] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.114039 24783 raft_consensus.cc:3037] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.116341 24783 raft_consensus.cc:2454] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:11.116377 24653 raft_consensus.cc:2454] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 1.
I20240502 21:27:11.116798 24456 leader_election.cc:302] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:11.117112 26181 raft_consensus.cc:2781] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.117280 26181 raft_consensus.cc:686] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:11.117578 26181 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } }
I20240502 21:27:11.121908 24240 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:11.406518 26107 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.415987 26195 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.475323 26197 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.484910 26086 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.524129 26181 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.529479 26181 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:11.559471 26200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:11.559832 26200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:11.567742 26200 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 269.689 ms
time per row: 0.134844 ms
I20240502 21:27:11.989076 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.5:42587
--local_ip_for_outbound_sockets=127.16.122.5
--tserver_master_addrs=127.16.122.62:45825
--webserver_port=36891
--webserver_interface=127.16.122.5
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:27:12.371860 26261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:12.372268 26261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:12.372429 26261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:12.381208 26261 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:12.381342 26261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:12.381492 26261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.5
I20240502 21:27:12.391372 26261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.5:42587
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.122.5
--webserver_port=36891
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.5
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:12.392767 26261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:12.395148 26261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:12.407582 26267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:12.407784 26266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:12.408303 26269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:12.408816 26261 server_base.cc:1034] running on GCE node
I20240502 21:27:12.409688 26261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:12.410753 26261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:12.412000 26261 hybrid_clock.cc:648] HybridClock initialized: now 1714685232411941 us; error 45 us; skew 500 ppm
I20240502 21:27:12.412407 26261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:12.415019 26261 webserver.cc:460] Webserver started at http://127.16.122.5:36891/ using document root <none> and password file <none>
I20240502 21:27:12.415791 26261 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:12.415937 26261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:12.422091 26261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 21:27:12.425637 26275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:12.427137 26261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:12.427348 26261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b77f8f52372241d483f457389fda0f93"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:27:12.428215 26261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:12.441850 26261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:12.442692 26261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:12.442972 26261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:12.443964 26261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:12.445560 26261 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:12.445641 26261 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:12.445734 26261 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:12.445778 26261 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:12.478175 26261 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.5:42587
I20240502 21:27:12.478219 26389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.5:42587 every 8 connection(s)
I20240502 21:27:12.479811 26261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:27:12.485337 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 26261
I20240502 21:27:12.485971 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.6:41751
--local_ip_for_outbound_sockets=127.16.122.6
--tserver_master_addrs=127.16.122.62:45825
--webserver_port=36403
--webserver_interface=127.16.122.6
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:12.495011 26390 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:27:12.495339 26390 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:12.496220 26390 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:27:12.498004 24246 ts_manager.cc:194] Re-registered known tserver with Master: b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:27:12.498921 24246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.5:47385
W20240502 21:27:12.854765 26394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:12.855073 26394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:12.855213 26394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:12.863734 26394 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:12.863879 26394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:12.864048 26394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.6
I20240502 21:27:12.873574 26394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.6:41751
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.122.6
--webserver_port=36403
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.6
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:12.874938 26394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:12.876752 26394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:12.891206 26399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:12.891204 26401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:12.891911 26403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:12.892413 26394 server_base.cc:1034] running on GCE node
I20240502 21:27:12.893031 26394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:12.894161 26394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:12.895452 26394 hybrid_clock.cc:648] HybridClock initialized: now 1714685232895384 us; error 52 us; skew 500 ppm
I20240502 21:27:12.895814 26394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:12.898624 26394 webserver.cc:460] Webserver started at http://127.16.122.6:36403/ using document root <none> and password file <none>
I20240502 21:27:12.899389 26394 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:12.899521 26394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:12.905349 26394 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20240502 21:27:12.908159 26409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:12.909533 26394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:12.909719 26394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "836775e329f148e5908a1afc54f06428"
format_stamp: "Formatted at 2024-05-02 21:26:52 on dist-test-slave-p9xn"
I20240502 21:27:12.910588 26394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:12.935395 26394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:12.936378 26394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:12.936691 26394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:12.937647 26394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:12.939306 26394 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:12.939379 26394 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:12.939471 26394 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:12.939515 26394 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:12.975919 26394 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.6:41751
I20240502 21:27:12.976008 26523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.6:41751 every 8 connection(s)
I20240502 21:27:12.977376 26394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:27:12.982295 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 26394
I20240502 21:27:12.982931 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.7:36311
--local_ip_for_outbound_sockets=127.16.122.7
--tserver_master_addrs=127.16.122.62:45825
--webserver_port=39651
--webserver_interface=127.16.122.7
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:12.990936 26524 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:27:12.991329 26524 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:12.992496 26524 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:27:12.994480 24246 ts_manager.cc:194] Re-registered known tserver with Master: 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751)
I20240502 21:27:12.995368 24246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.6:53209
W20240502 21:27:13.358275 26527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:13.358644 26527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:13.358798 26527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:13.366624 26527 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:13.366770 26527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:13.366930 26527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.7
I20240502 21:27:13.376178 26527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.7:36311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.122.7
--webserver_port=39651
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.7
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:13.377882 26527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:13.380100 26527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:13.393077 26532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:13.393662 26533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:13.393899 26535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:13.394599 26527 server_base.cc:1034] running on GCE node
I20240502 21:27:13.395220 26527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:13.396261 26527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:13.397471 26527 hybrid_clock.cc:648] HybridClock initialized: now 1714685233397428 us; error 45 us; skew 500 ppm
I20240502 21:27:13.397838 26527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:13.400274 26527 webserver.cc:460] Webserver started at http://127.16.122.7:39651/ using document root <none> and password file <none>
I20240502 21:27:13.401043 26527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:13.401182 26527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:13.407897 26527 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:27:13.410979 26541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:13.412487 26527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 21:27:13.412690 26527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9b0952b24bd8400c9940dbd5e28417b4"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:27:13.413627 26527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:13.441879 26527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:13.442983 26527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:13.443301 26527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:13.444286 26527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:13.445863 26527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:13.445935 26527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:13.446031 26527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:13.446074 26527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:13.477870 26527 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.7:36311
I20240502 21:27:13.477946 26656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.7:36311 every 8 connection(s)
I20240502 21:27:13.479506 26527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:27:13.489236 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 26527
I20240502 21:27:13.489858 16872 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
/tmp/dist-test-task3Dzs1I/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.8:37045
--local_ip_for_outbound_sockets=127.16.122.8
--tserver_master_addrs=127.16.122.62:45825
--webserver_port=34171
--webserver_interface=127.16.122.8
--builtin_ntp_servers=127.16.122.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:13.493780 26657 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:27:13.494206 26657 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:13.495254 26657 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:27:13.496989 24246 ts_manager.cc:194] Re-registered known tserver with Master: 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
I20240502 21:27:13.498216 24246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.7:58031
I20240502 21:27:13.501531 26390 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
W20240502 21:27:13.866513 26660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:13.866811 26660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:13.866946 26660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:13.874554 26660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:13.874680 26660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:13.874843 26660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.8
I20240502 21:27:13.884886 26660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.8:37045
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.122.8
--webserver_port=34171
--tserver_master_addrs=127.16.122.62:45825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.8
--log_dir=/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:13.886271 26660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:13.888080 26660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:13.900578 26665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:13.901041 26668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:13.901392 26666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:13.902078 26660 server_base.cc:1034] running on GCE node
I20240502 21:27:13.902741 26660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:13.903741 26660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:13.904937 26660 hybrid_clock.cc:648] HybridClock initialized: now 1714685233904886 us; error 46 us; skew 500 ppm
I20240502 21:27:13.905257 26660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:13.907617 26660 webserver.cc:460] Webserver started at http://127.16.122.8:34171/ using document root <none> and password file <none>
I20240502 21:27:13.908425 26660 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:13.908573 26660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:13.915011 26660 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 21:27:13.918118 26674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:13.919567 26660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:13.919739 26660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1769ee000eb24298a367a0de1209e6e0"
format_stamp: "Formatted at 2024-05-02 21:26:53 on dist-test-slave-p9xn"
I20240502 21:27:13.920560 26660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:13.947729 26660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:13.948743 26660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:13.949065 26660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:13.950103 26660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:13.951728 26660 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:13.951802 26660 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:13.951948 26660 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:13.951997 26660 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:13.993062 26660 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.8:37045
I20240502 21:27:13.993100 26790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.8:37045 every 8 connection(s)
I20240502 21:27:13.995096 26660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:27:13.998539 26524 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
I20240502 21:27:13.999845 16872 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu as pid 26660
I20240502 21:27:14.012480 26791 heartbeater.cc:347] Connected to a master server at 127.16.122.62:45825
I20240502 21:27:14.012810 26791 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:14.014688 26791 heartbeater.cc:511] Master 127.16.122.62:45825 requested a full tablet report, sending...
I20240502 21:27:14.016396 24246 ts_manager.cc:194] Re-registered known tserver with Master: 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045)
I20240502 21:27:14.017370 24246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.8:39073
I20240502 21:27:14.500990 26657 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
I20240502 21:27:15.022650 26791 heartbeater.cc:503] Master 127.16.122.62:45825 was elected leader, sending a full tablet report...
I20240502 21:27:15.303336 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.370175 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.370718 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.388949 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.445742 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:15.486069 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.513885 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.534565 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:15.607416 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.626442 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.641839 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:15.664460 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.677423 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.697643 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.700837 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.734171 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:15.769819 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.813652 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:15.853675 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:15.865723 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.868922 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:15.881649 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.907229 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:15.913043 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.135143 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:16.169454 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.179910 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:16.204231 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.218403 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.256237 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:16.257860 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.316635 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:16.415753 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:16.435933 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:16.484830 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.499228 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:16.505098 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.550259 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.551684 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:16.587323 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.081395 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.103888 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.120731 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.161047 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.235101 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.272159 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.291339 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.304771 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.423120 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.441174 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.550778 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.555500 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.561213 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.581066 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.586066 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.610620 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.625656 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.636715 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.643349 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.657925 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.662108 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.665764 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.694082 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:17.696702 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.701234 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.722764 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.726537 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.750785 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.754199 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.783900 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:17.795357 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:17.854228 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:18.221247 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.268496 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:18.295468 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.317543 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:18.322909 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:18.344656 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.403818 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:18.417508 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.763088 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:18.768805 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:18.774770 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.810950 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.830976 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:18.856457 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.859644 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:18.897302 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 21:27:18.905150 24587 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.1:36473, user_credentials={real_user=slave}} blocked reactor thread for 53497.6us
I20240502 21:27:19.135605 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:19.177309 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.216835 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:19.231300 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:19.259063 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.305948 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.340127 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.346447 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:19.554749 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:19.561751 24784 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:19.569272 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.576669 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.605512 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:19.628829 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.634150 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:19.650959 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.669564 24522 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:19.670639 27361 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:19.677856 24652 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:19.678556 27361 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:19.694070 27361 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:19.698846 24652 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:19.707849 24522 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:19.715530 24243 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:19.731856 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 21:27:19.793686 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.2:43159, user_credentials={real_user=slave}} blocked reactor thread for 80847.2us
W20240502 21:27:19.798106 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e847a796029d4edea79694f5402a291a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:19.817327 24522 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:19.829704 24388 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:19.833223 27422 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:19.835091 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:19.847546 24652 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:19.848402 27422 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:19.859755 27273 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:19.862725 24653 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:19.881060 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:19.899240 24390 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:19.906502 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.915100 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.928586 24243 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:19.967234 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:19.967866 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 21:27:19.970049 24455 consensus_peers.cc:485] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc717a1b12e4380a4107e6deae2b837. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:19.975608 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:20.000604 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.098213 24390 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:20.113488 24522 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.114185 27274 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.125285 24653 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.126039 27274 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.154376 27570 ts_tablet_manager.cc:902] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:20.184454 27570 tablet_copy_client.cc:308] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:20.183888 27572 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.189550 24653 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.199802 24521 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.206593 24520 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:20.211890 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet e847a796029d4edea79694f5402a291a from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:45415)
I20240502 21:27:20.212224 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet e847a796029d4edea79694f5402a291a from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:45415: session id = b77f8f52372241d483f457389fda0f93-e847a796029d4edea79694f5402a291a
I20240502 21:27:20.212881 24247 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.217618 24804 tablet_copy_source_session.cc:205] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:20.225756 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e847a796029d4edea79694f5402a291a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:20.227550 24784 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.227805 24390 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.230412 27273 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.231290 27422 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:20.238711 24456 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.243990 27422 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.252333 24390 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.254093 24242 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:20.271950 24784 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.273029 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 21:27:20.279124 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d8a25580bc74004b0e31c34f841d2cc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.348201 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.363456 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:20.383522 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.399816 24520 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:20.407215 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.408064 27613 ts_tablet_manager.cc:902] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:20.413710 27570 tablet_copy_client.cc:783] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:20.416971 27570 tablet_copy_client.cc:647] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:20.422477 27570 tablet_copy_client.cc:523] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:20.431744 24653 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.433082 27273 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.435814 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:20.437292 27273 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:20.442404 24653 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:20.449601 24242 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:20.450600 27570 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:20.459354 24784 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.460143 27422 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.477731 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 21:27:20.480895 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e5a87719c2544a5b0cb591ed9f46df8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.493391 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:20.510123 24784 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:20.572082 27613 tablet_copy_client.cc:308] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:20.596875 24653 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:20.598083 27570 log.cc:824] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:20.620510 24784 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.621405 24390 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.624264 27548 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.625645 27548 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.647250 24390 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.650956 27548 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.655570 24242 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.657564 24784 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.675194 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 0fc717a1b12e4380a4107e6deae2b837 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:34905)
I20240502 21:27:20.675629 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 0fc717a1b12e4380a4107e6deae2b837 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:34905: session id = 1769ee000eb24298a367a0de1209e6e0-0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:20.683741 24542 tablet_copy_source_session.cc:205] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:20.691557 27613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:27:20.698166 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049bec57d2474c328d2e26e16a4d33a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.699482 24390 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:20.710865 24520 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:20.716449 24388 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:27:20.723834 24456 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.724261 27273 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:20.735707 27422 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.739554 24388 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.750262 27613 tablet_copy_client.cc:783] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:20.750759 27613 tablet_copy_client.cc:647] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
W20240502 21:27:20.752838 24456 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.755344 27613 tablet_copy_client.cc:523] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:20.755695 24389 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:20.763890 24247 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.768978 27656 ts_tablet_manager.cc:902] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:20.779562 24784 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:20.780530 24323 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b6a4a74e0544e0e99e734a145c91d3b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.781103 24784 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:20.781019 27586 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.781953 27273 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:20.784871 24783 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.785843 24520 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.786446 27572 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.788480 27613 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:20.801283 27586 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.802186 24783 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.811508 24243 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.838140 24784 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.839140 27572 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.843343 24653 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:20.844231 27572 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:20.849777 27570 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:20.850615 27570 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:20.851238 27570 ts_tablet_manager.cc:1367] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.401s user 0.113s sys 0.039s
I20240502 21:27:20.851820 27586 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:20.851940 24784 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:20.858529 27570 raft_consensus.cc:348] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:20.859134 27570 raft_consensus.cc:729] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:20.859256 24242 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:20.859795 27570 consensus_queue.cc:260] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:20.877416 27570 ts_tablet_manager.cc:1398] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.026s user 0.012s sys 0.005s
I20240502 21:27:20.878657 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-e847a796029d4edea79694f5402a291a received from {username='slave'} at 127.16.122.5:45415
I20240502 21:27:20.878863 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session b77f8f52372241d483f457389fda0f93-e847a796029d4edea79694f5402a291a on tablet e847a796029d4edea79694f5402a291a with peer b77f8f52372241d483f457389fda0f93
W20240502 21:27:20.889600 24322 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bf18ffb96d04d5e9ec8ee9563812ba0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.893481 24520 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:20.892899 24653 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:20.897943 27613 log.cc:824] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:20.915938 27656 tablet_copy_client.cc:308] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:20.940778 24390 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:20.945856 24652 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:27:20.946750 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d8a25580bc74004b0e31c34f841d2cc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.947158 27572 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:20.951333 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d8a25580bc74004b0e31c34f841d2cc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:20.951495 27572 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.954273 24652 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.964802 27570 ts_tablet_manager.cc:902] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:20.977226 24242 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:20.994167 24520 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:20.995600 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:20.995939 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:20.996554 27687 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.000334 24410 tablet_copy_source_session.cc:205] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.007354 24520 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:21.009709 24390 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:21.017474 24784 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:21.022722 27656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.023499 24520 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.024466 27687 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:21.026077 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98b5668e0d504db5a7874f041c14dbfa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.028218 24653 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.029023 27703 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.033030 27586 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:21.033958 24653 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:21.041687 24244 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.047782 24522 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:21.075376 24520 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.076702 27508 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.079335 24390 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:21.081876 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff7e019fa82d4648afaf130d3f6343d6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.084681 27632 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.088316 27508 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.092000 24520 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.099035 24390 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.120659 24242 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.122371 27570 tablet_copy_client.cc:308] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:21.176770 24520 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:21.182883 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:21.184064 26345 raft_consensus.cc:1203] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:21.185215 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:21.209266 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:21.210814 24455 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9963dcea478749e0a6e076fac6537fc4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.211493 24653 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.212435 27657 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.219612 24784 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.220484 27657 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.227381 27740 ts_tablet_manager.cc:902] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:21.240077 27422 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.243288 24652 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.244024 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:21.245524 24784 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.248546 24247 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.264561 27656 tablet_copy_client.cc:783] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.265125 27656 tablet_copy_client.cc:647] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.265254 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 2e5a87719c2544a5b0cb591ed9f46df8 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:21.265482 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 2e5a87719c2544a5b0cb591ed9f46df8 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-2e5a87719c2544a5b0cb591ed9f46df8
I20240502 21:27:21.268510 24542 tablet_copy_source_session.cc:205] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.271142 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5a87719c2544a5b0cb591ed9f46df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.275408 27656 tablet_copy_client.cc:523] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.277448 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:21.278164 27753 ts_tablet_manager.cc:902] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:21.283288 27570 tablet_copy_client.cc:783] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.292820 27570 tablet_copy_client.cc:647] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.297200 27570 tablet_copy_client.cc:523] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.298708 27739 ts_tablet_manager.cc:902] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:21.300237 24784 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:21.303158 27740 tablet_copy_client.cc:308] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:21.313895 27570 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:21.317271 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:21.329182 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b2661e0b5fe4a0b94c150b14b8fd9ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.330173 24522 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.331022 27632 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.334579 24390 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.335122 27656 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:21.335191 27632 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.338079 27624 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.340942 24522 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.353183 24240 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.381933 27750 ts_tablet_manager.cc:902] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:21.387387 24390 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:21.395413 27753 tablet_copy_client.cc:308] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:21.396183 27777 ts_tablet_manager.cc:902] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:21.399138 24652 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:21.402750 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:21.420096 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 049bec57d2474c328d2e26e16a4d33a9 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:46751)
I20240502 21:27:21.420624 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 049bec57d2474c328d2e26e16a4d33a9 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:46751: session id = 836775e329f148e5908a1afc54f06428-049bec57d2474c328d2e26e16a4d33a9
I20240502 21:27:21.426873 24673 tablet_copy_source_session.cc:205] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.433918 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049bec57d2474c328d2e26e16a4d33a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.434134 24389 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.436400 24784 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.438490 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.439611 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.445653 27739 tablet_copy_client.cc:308] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:21.457182 27750 tablet_copy_client.cc:308] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:21.474864 27637 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:21.479300 27777 tablet_copy_client.cc:308] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:21.480306 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 2b6a4a74e0544e0e99e734a145c91d3b from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:21.480526 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 2b6a4a74e0544e0e99e734a145c91d3b from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-2b6a4a74e0544e0e99e734a145c91d3b
I20240502 21:27:21.483667 24410 tablet_copy_source_session.cc:205] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.485601 27777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6a4a74e0544e0e99e734a145c91d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.490820 24784 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:21.489084 24240 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.506917 27740 tablet_copy_client.cc:783] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.507439 27740 tablet_copy_client.cc:647] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.509027 27656 log.cc:824] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:21.511540 27809 ts_tablet_manager.cc:902] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:21.515480 27740 tablet_copy_client.cc:523] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.517463 27777 tablet_copy_client.cc:783] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.517815 27777 tablet_copy_client.cc:647] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.520442 27777 tablet_copy_client.cc:523] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.521450 27740 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:21.525815 27777 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:21.540305 27740 log.cc:824] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:21.541110 27814 ts_tablet_manager.cc:902] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:21.555284 27809 tablet_copy_client.cc:308] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:21.556461 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 98b5668e0d504db5a7874f041c14dbfa from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:21.556668 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 98b5668e0d504db5a7874f041c14dbfa from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-98b5668e0d504db5a7874f041c14dbfa
I20240502 21:27:21.560021 24410 tablet_copy_source_session.cc:205] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.561937 27613 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:21.562445 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.562889 27613 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:21.563498 27613 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.777s user 0.080s sys 0.000s
I20240502 21:27:21.573539 27570 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:21.576197 27814 tablet_copy_client.cc:308] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:21.586421 27809 tablet_copy_client.cc:783] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.587325 27570 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:21.587700 27570 ts_tablet_manager.cc:1367] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.274s user 0.055s sys 0.014s
I20240502 21:27:21.588973 27570 raft_consensus.cc:348] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:21.589262 27570 raft_consensus.cc:729] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:21.589309 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet ac35b080037646b4b8b34c0eab2de4e6 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:21.589511 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet ac35b080037646b4b8b34c0eab2de4e6 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:21.589493 27570 consensus_queue.cc:260] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:21.589617 24522 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:21.593590 24542 tablet_copy_source_session.cc:205] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.594065 27570 ts_tablet_manager.cc:1398] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 21:27:21.598022 27809 tablet_copy_client.cc:647] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.599030 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-2e5a87719c2544a5b0cb591ed9f46df8 received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:21.599249 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-2e5a87719c2544a5b0cb591ed9f46df8 on tablet 2e5a87719c2544a5b0cb591ed9f46df8 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:21.600515 27809 tablet_copy_client.cc:523] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.606639 27809 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:21.608911 24652 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:21.618904 24653 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.619639 27422 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.623816 27740 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:21.624657 27740 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:21.625411 27740 ts_tablet_manager.cc:1367] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.104s user 0.068s sys 0.010s
I20240502 21:27:21.632982 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet ac35b080037646b4b8b34c0eab2de4e6 from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:35031)
I20240502 21:27:21.633230 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet ac35b080037646b4b8b34c0eab2de4e6 from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:35031: session id = 9b0952b24bd8400c9940dbd5e28417b4-ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:21.634397 27740 raft_consensus.cc:348] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:21.635180 27740 raft_consensus.cc:729] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:21.635907 27740 consensus_queue.cc:260] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:21.636454 24542 tablet_copy_source_session.cc:205] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.640981 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:21.645279 27740 ts_tablet_manager.cc:1398] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.020s user 0.006s sys 0.009s
I20240502 21:27:21.649971 27753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b080037646b4b8b34c0eab2de4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.650092 27570 ts_tablet_manager.cc:902] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:21.659162 27753 tablet_copy_client.cc:783] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.659624 27753 tablet_copy_client.cc:647] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
W20240502 21:27:21.668488 24455 consensus_peers.cc:485] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.687281 27570 tablet_copy_client.cc:308] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:21.688278 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet ff7e019fa82d4648afaf130d3f6343d6 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:45415)
I20240502 21:27:21.688552 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet ff7e019fa82d4648afaf130d3f6343d6 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:45415: session id = b77f8f52372241d483f457389fda0f93-ff7e019fa82d4648afaf130d3f6343d6
I20240502 21:27:21.689392 24389 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:21.692028 24804 tablet_copy_source_session.cc:205] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.693850 27570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e019fa82d4648afaf130d3f6343d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.695068 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:21.695984 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:21.699625 26479 raft_consensus.cc:1203] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:21.702760 24410 tablet_copy_source_session.cc:205] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.707391 27750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.710448 27422 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:21.712496 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-049bec57d2474c328d2e26e16a4d33a9 received from {username='slave'} at 127.16.122.6:46751
I20240502 21:27:21.713541 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 836775e329f148e5908a1afc54f06428-049bec57d2474c328d2e26e16a4d33a9 on tablet 049bec57d2474c328d2e26e16a4d33a9 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:21.713537 27570 tablet_copy_client.cc:783] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.713877 27570 tablet_copy_client.cc:647] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.714632 24651 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
W20240502 21:27:21.724385 24586 consensus_peers.cc:485] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566929e603d54be4bf931413f0fa1d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.725721 24242 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.727907 24390 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.729532 27637 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:21.731305 24586 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc5c40e2bf0340958ecd3e7541209e95. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.739507 27570 tablet_copy_client.cc:523] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.741338 27624 raft_consensus.cc:1050] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:21.743775 27624 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:21.749392 27570 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:21.752534 24390 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.755719 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b080037646b4b8b34c0eab2de4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.763037 27796 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:21.768723 24388 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:21.771720 24242 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:21.782403 24520 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:21.786307 27796 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:21.790597 26345 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.791723 27632 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.795460 27739 tablet_copy_client.cc:783] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.795835 27739 tablet_copy_client.cc:647] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.799518 24522 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.800447 27624 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.807221 24652 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.807518 27777 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:21.808008 27624 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.808213 27777 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:21.808897 27777 ts_tablet_manager.cc:1367] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.283s user 0.060s sys 0.007s
I20240502 21:27:21.814244 24520 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:21.817041 27632 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
W20240502 21:27:21.818400 24455 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9963dcea478749e0a6e076fac6537fc4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.819716 24651 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.820788 27657 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.822921 27273 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:21.823879 24456 consensus_peers.cc:485] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc717a1b12e4380a4107e6deae2b837. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:21.824455 27657 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:21.825052 24520 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.829149 24783 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:21.831215 27750 tablet_copy_client.cc:783] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.831549 27750 tablet_copy_client.cc:647] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.832032 27662 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:21.833667 24651 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:21.834393 27753 tablet_copy_client.cc:523] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.841977 24244 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:21.842844 27750 tablet_copy_client.cc:523] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.848861 24247 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.851231 24653 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.852602 27753 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:21.845674 26345 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.855404 24390 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:21.861431 27777 raft_consensus.cc:348] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:21.863294 27777 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:21.863976 27777 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:21.867683 27750 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:21.873742 24782 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:21.875684 27809 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:21.876238 27809 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:21.876545 27809 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.271s user 0.051s sys 0.011s
I20240502 21:27:21.878816 26343 raft_consensus.cc:1203] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:21.881038 27809 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:21.881337 27809 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:21.881564 27809 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:21.882699 26343 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:21.884636 27846 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:27:21.891731 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9963dcea478749e0a6e076fac6537fc4. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:21.891981 24456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.6:41751, user_credentials={real_user=slave}} blocked reactor thread for 59917.9us
I20240502 21:27:21.893462 27656 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:21.894053 27656 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:21.893967 24651 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:21.894440 27656 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.564s user 0.064s sys 0.006s
I20240502 21:27:21.894935 27632 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.895357 27656 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:21.895645 27656 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:21.895901 27656 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:21.896044 27624 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:21.896998 27809 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20240502 21:27:21.897580 26343 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.898540 24653 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:21.900069 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e847a796029d4edea79694f5402a291a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:21.901990 24243 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:21.903653 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-98b5668e0d504db5a7874f041c14dbfa received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:21.903885 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-98b5668e0d504db5a7874f041c14dbfa on tablet 98b5668e0d504db5a7874f041c14dbfa with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:21.907984 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet e847a796029d4edea79694f5402a291a with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:21.914789 24784 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
W20240502 21:27:21.916613 27828 log.cc:924] Time spent T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20240502 21:27:21.938341 27656 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20240502 21:27:21.944252 26612 raft_consensus.cc:1203] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:21.946830 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-9d8a25580bc74004b0e31c34f841d2cc received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:21.947055 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-9d8a25580bc74004b0e31c34f841d2cc on tablet 9d8a25580bc74004b0e31c34f841d2cc with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:21.949710 27739 tablet_copy_client.cc:523] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.957636 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:45315)
I20240502 21:27:21.957940 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:45315: session id = 836775e329f148e5908a1afc54f06428-9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:21.971352 27739 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:21.971503 27843 ts_tablet_manager.cc:902] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:21.972077 27852 tablet_replica.cc:330] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:21.972957 27852 raft_consensus.cc:2229] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:21.973485 27852 raft_consensus.cc:2258] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:21.974205 27853 ts_tablet_manager.cc:902] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:21.975808 24410 tablet_copy_source_session.cc:205] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:21.977713 27814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:21.986836 27814 tablet_copy_client.cc:783] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:21.987322 27814 tablet_copy_client.cc:647] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:21.990458 27814 tablet_copy_client.cc:523] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:21.990656 27777 ts_tablet_manager.cc:1398] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.182s user 0.010s sys 0.016s
I20240502 21:27:21.992282 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-2b6a4a74e0544e0e99e734a145c91d3b received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:21.992477 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-2b6a4a74e0544e0e99e734a145c91d3b on tablet 2b6a4a74e0544e0e99e734a145c91d3b with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:21.997998 27851 ts_tablet_manager.cc:902] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:22.005110 26612 raft_consensus.cc:1203] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:27:22.012730 27851 tablet_copy_client.cc:308] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:22.019416 27613 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:22.020216 27613 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:22.020882 27613 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:22.021575 24522 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:22.024426 27843 tablet_copy_client.cc:308] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
W20240502 21:27:22.021569 24456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:33687, user_credentials={real_user=slave}} blocked reactor thread for 63765.9us
I20240502 21:27:22.025444 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 9963dcea478749e0a6e076fac6537fc4 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:22.025631 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 9963dcea478749e0a6e076fac6537fc4 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:22.029070 24542 tablet_copy_source_session.cc:205] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.030817 27843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.031453 27740 ts_tablet_manager.cc:902] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:22.032753 27852 ts_tablet_manager.cc:1875] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:22.039408 27814 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:22.061048 27843 tablet_copy_client.cc:783] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.061508 27843 tablet_copy_client.cc:647] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.063814 27843 tablet_copy_client.cc:523] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:22.067711 27843 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:22.069777 27613 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.506s user 0.010s sys 0.002s
I20240502 21:27:22.070734 27570 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:22.073518 24389 consensus_queue.cc:237] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
W20240502 21:27:22.078078 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adde18faca794600bb4aa24f0cf744e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:22.078436 24784 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:22.079849 27613 ts_tablet_manager.cc:1430] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Tablet startup took 516ms
I20240502 21:27:22.081102 27703 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:22.084868 27570 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:22.085255 27570 ts_tablet_manager.cc:1367] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.336s user 0.050s sys 0.008s
I20240502 21:27:22.086428 27570 raft_consensus.cc:348] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:22.086711 27570 raft_consensus.cc:729] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:22.088640 27570 consensus_queue.cc:260] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:22.091107 27570 ts_tablet_manager.cc:1398] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:22.092104 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-ff7e019fa82d4648afaf130d3f6343d6 received from {username='slave'} at 127.16.122.5:45415
I20240502 21:27:22.092264 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session b77f8f52372241d483f457389fda0f93-ff7e019fa82d4648afaf130d3f6343d6 on tablet ff7e019fa82d4648afaf130d3f6343d6 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:22.101200 24652 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:22.080914 27613 ts_tablet_manager.cc:1432] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Trace:
0502 21:27:20.389878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:20.389977 (+ 99us) service_pool.cc:225] Handling call
0502 21:27:20.411458 (+ 21481us) ts_tablet_manager.cc:903] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
0502 21:27:20.686284 (+274826us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:20.703200 (+ 16916us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:20.704148 (+ 948us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:27:20.744727 (+ 40579us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:20.745325 (+ 598us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:20.746047 (+ 722us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:20.746117 (+ 70us) inbound_call.cc:173] Queueing success response
0502 21:27:20.781974 (+ 35857us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 21:27:21.563633 (+781659us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 21:27:21.564012 (+ 379us) tablet_replica.cc:259] Starting instrumentation
0502 21:27:21.564358 (+ 346us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":121,"raft.queue_time_us":24149,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":1593,"tablet-copy.queue_time_us":17228,"tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us":1491,"tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us":1006,"tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us":2748,"thread_start_us":222484,"threads_started":6}
I20240502 21:27:22.102137 27586 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:22.103894 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-0fc717a1b12e4380a4107e6deae2b837 received from {username='slave'} at 127.16.122.8:34905
I20240502 21:27:22.104096 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-0fc717a1b12e4380a4107e6deae2b837 on tablet 0fc717a1b12e4380a4107e6deae2b837 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:22.109153 27853 tablet_copy_client.cc:308] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:22.109884 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 566929e603d54be4bf931413f0fa1d7d from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:38365)
I20240502 21:27:22.110100 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 566929e603d54be4bf931413f0fa1d7d from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:38365: session id = 1769ee000eb24298a367a0de1209e6e0-566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:22.110448 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:45415)
I20240502 21:27:22.110615 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:45415: session id = b77f8f52372241d483f457389fda0f93-7b2661e0b5fe4a0b94c150b14b8fd9ec
I20240502 21:27:22.113363 24673 tablet_copy_source_session.cc:205] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.113585 24804 tablet_copy_source_session.cc:205] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.120345 27851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.124658 24782 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:22.129778 27613 ts_tablet_manager.cc:902] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:22.141849 24653 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:22.142560 24522 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:22.142657 27632 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:22.143221 27846 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:22.147897 27586 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:22.151000 24389 consensus_queue.cc:237] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:22.153808 27852 ts_tablet_manager.cc:1888] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:22.155200 27853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.164393 24651 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:22.165051 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 855b247f654d41c2bbb13abb25d35c1c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:22.165741 27703 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:22.173290 24242 catalog_manager.cc:5484] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:22.175297 24782 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:22.177186 24784 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:22.178507 27586 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:22.181378 27703 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.185458 27852 log.cc:1196] T e847a796029d4edea79694f5402a291a P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/e847a796029d4edea79694f5402a291a
I20240502 21:27:22.187043 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet e847a796029d4edea79694f5402a291a (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:22.186766 24650 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.212253 27853 tablet_copy_client.cc:783] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.213368 27853 tablet_copy_client.cc:647] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.215883 27853 tablet_copy_client.cc:523] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:22.219539 27846 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.221899 27853 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:22.195777 24782 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.226183 26342 raft_consensus.cc:1203] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:22.225561 24652 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:22.229125 24522 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.260224 24247 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.255154 26742 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.264286 27843 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:22.264820 27843 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:22.265118 27843 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.197s user 0.069s sys 0.026s
I20240502 21:27:22.265995 27843 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:22.266264 27843 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:22.266516 27843 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:22.268374 27843 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:22.269275 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-9963dcea478749e0a6e076fac6537fc4 received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:22.269436 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-9963dcea478749e0a6e076fac6537fc4 on tablet 9963dcea478749e0a6e076fac6537fc4 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:22.274487 27740 tablet_copy_client.cc:308] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:22.275490 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 0fc717a1b12e4380a4107e6deae2b837 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:22.275758 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 0fc717a1b12e4380a4107e6deae2b837 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:22.279428 24542 tablet_copy_source_session.cc:205] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.282610 27915 ts_tablet_manager.cc:902] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:22.285241 27613 tablet_copy_client.cc:308] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:22.286487 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet dc5c40e2bf0340958ecd3e7541209e95 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:38365)
I20240502 21:27:22.286734 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet dc5c40e2bf0340958ecd3e7541209e95 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:38365: session id = 1769ee000eb24298a367a0de1209e6e0-dc5c40e2bf0340958ecd3e7541209e95
I20240502 21:27:22.291919 24673 tablet_copy_source_session.cc:205] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 21:27:22.299463 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.2:43159, user_credentials={real_user=slave}} blocked reactor thread for 64158.7us
I20240502 21:27:22.312208 27853 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:22.312773 27853 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:22.313118 27853 ts_tablet_manager.cc:1367] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.091s user 0.055s sys 0.000s
I20240502 21:27:22.313892 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.313997 27853 raft_consensus.cc:348] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:22.314288 27853 raft_consensus.cc:729] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:22.314559 27853 consensus_queue.cc:260] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:22.316201 27853 ts_tablet_manager.cc:1398] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 21:27:22.262686 24242 catalog_manager.cc:5484] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.320602 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-7b2661e0b5fe4a0b94c150b14b8fd9ec received from {username='slave'} at 127.16.122.5:45415
I20240502 21:27:22.320816 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session b77f8f52372241d483f457389fda0f93-7b2661e0b5fe4a0b94c150b14b8fd9ec on tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:22.322764 27853 tablet_replica.cc:330] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: stopping tablet replica
I20240502 21:27:22.323417 27853 raft_consensus.cc:2229] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 21:27:22.323676 27851 tablet_copy_client.cc:783] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.323740 27853 raft_consensus.cc:2258] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 21:27:22.324137 27851 tablet_copy_client.cc:647] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.325927 27853 ts_tablet_manager.cc:1875] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:22.334544 27853 ts_tablet_manager.cc:1888] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:22.334776 27853 log.cc:1196] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/wals/9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:22.335306 27853 ts_tablet_manager.cc:902] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:22.343417 24650 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:22.350159 27613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5c40e2bf0340958ecd3e7541209e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.354703 27855 raft_consensus.cc:1050] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:22.355805 27855 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:22.367993 27637 raft_consensus.cc:1050] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:22.369231 27637 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:22.372875 27851 tablet_copy_client.cc:523] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:22.374300 27703 raft_consensus.cc:1050] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:22.375438 27703 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
W20240502 21:27:22.381618 24718 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a35079a106c4cca94cf65ffa93aefbf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:22.393565 27851 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:22.399075 27687 raft_consensus.cc:1050] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:22.400180 27687 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:22.426637 27613 tablet_copy_client.cc:783] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.427078 27613 tablet_copy_client.cc:647] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.429785 27613 tablet_copy_client.cc:523] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:22.443740 26344 raft_consensus.cc:1203] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:22.449993 27613 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:22.469660 24652 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:22.515982 26344 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.516860 27662 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.527720 26479 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.530687 24653 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.531046 24784 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.531757 27662 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.532652 27657 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.550714 26611 raft_consensus.cc:1203] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:22.556975 27814 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:22.557626 27814 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:22.558081 27814 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.519s user 0.076s sys 0.002s
I20240502 21:27:22.565388 24389 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.567034 24782 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.581120 27953 ts_tablet_manager.cc:902] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:22.583757 27422 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
W20240502 21:27:22.587386 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:22.593772 24782 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.594489 27851 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:22.594190 24241 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:22.594986 27851 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:22.595305 27851 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.202s user 0.055s sys 0.006s
I20240502 21:27:22.596410 27851 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:22.596666 27851 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:22.596861 27851 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:22.601210 26612 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.601744 27966 ts_tablet_manager.cc:902] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:22.602710 24653 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.605558 27851 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 21:27:22.606379 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-566929e603d54be4bf931413f0fa1d7d received from {username='slave'} at 127.16.122.8:38365
I20240502 21:27:22.606561 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-566929e603d54be4bf931413f0fa1d7d on tablet 566929e603d54be4bf931413f0fa1d7d with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:22.607386 27958 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.621363 27927 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.623404 27927 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.625005 27927 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.625406 27814 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:22.627851 27814 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:22.632129 27814 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:22.639353 27966 tablet_copy_client.cc:308] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:22.640357 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 855b247f654d41c2bbb13abb25d35c1c from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:22.640596 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 855b247f654d41c2bbb13abb25d35c1c from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-855b247f654d41c2bbb13abb25d35c1c
I20240502 21:27:22.650384 24410 tablet_copy_source_session.cc:205] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.636593 27915 tablet_copy_client.cc:308] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:22.652541 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 9963dcea478749e0a6e076fac6537fc4 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:22.652771 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 9963dcea478749e0a6e076fac6537fc4 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:22.657621 24542 tablet_copy_source_session.cc:205] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.658969 27814 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.101s user 0.007s sys 0.009s
I20240502 21:27:22.659667 27966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855b247f654d41c2bbb13abb25d35c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.660663 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-9d8a25580bc74004b0e31c34f841d2cc received from {username='slave'} at 127.16.122.6:45315
I20240502 21:27:22.660825 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 836775e329f148e5908a1afc54f06428-9d8a25580bc74004b0e31c34f841d2cc on tablet 9d8a25580bc74004b0e31c34f841d2cc with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:22.661895 27953 tablet_copy_client.cc:308] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:22.685968 27966 tablet_copy_client.cc:783] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.686758 27966 tablet_copy_client.cc:647] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.694275 27851 ts_tablet_manager.cc:902] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:22.696125 27966 tablet_copy_client.cc:523] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:22.709064 27966 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:22.729846 27915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.741465 27662 raft_consensus.cc:1050] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:22.742797 27662 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
W20240502 21:27:22.749569 24456 consensus_peers.cc:485] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc717a1b12e4380a4107e6deae2b837. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:22.751001 24650 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.751896 27851 tablet_copy_client.cc:308] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:22.751827 27422 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.752703 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet adde18faca794600bb4aa24f0cf744e6 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:22.752874 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet adde18faca794600bb4aa24f0cf744e6 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-adde18faca794600bb4aa24f0cf744e6
I20240502 21:27:22.752981 24390 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.754845 27422 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.755764 24410 tablet_copy_source_session.cc:205] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:22.756966 26742 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.758289 27657 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.759025 27657 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.760562 27851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde18faca794600bb4aa24f0cf744e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.762507 24390 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.763201 26742 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.769665 24247 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 5 to 6, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.770856 24650 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.791519 26612 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.791756 24652 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.792594 27586 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.794138 27952 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.801569 27851 tablet_copy_client.cc:783] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:22.801874 27851 tablet_copy_client.cc:647] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:22.809689 24652 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.811118 27851 tablet_copy_client.cc:523] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:22.813644 27952 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.818223 24522 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:22.819399 27952 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:22.822300 27851 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:22.836634 27586 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:22.837706 24652 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:22.843577 26610 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:22.857331 24247 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:22.862004 27966 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:22.862737 27966 tablet_bootstrap.cc:492] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:22.863266 27966 ts_tablet_manager.cc:1367] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.154s user 0.066s sys 0.010s
I20240502 21:27:22.864488 27966 raft_consensus.cc:348] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:22.866536 27966 raft_consensus.cc:729] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:22.866273 24390 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:22.866874 27966 consensus_queue.cc:260] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:22.872910 26610 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.873647 27703 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.880309 27586 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:22.888984 24650 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.890666 27963 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.891989 26344 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.892448 24650 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:22.894579 24520 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:22.897198 24782 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.897987 27657 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.901458 26342 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.902175 24247 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:22.903375 27657 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.907135 27657 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.911298 26342 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.918403 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e5a87719c2544a5b0cb591ed9f46df8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:22.919956 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b5668e0d504db5a7874f041c14dbfa with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:22.919757 24782 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.930953 24240 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:22.931458 24520 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:22.932302 27963 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:22.936897 24521 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:22.950906 24784 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:22.953018 27963 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.961957 24650 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:22.966513 24240 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:22.969666 24520 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
W20240502 21:27:22.971355 24587 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.1:36473, user_credentials={real_user=slave}} blocked reactor thread for 64472.4us
I20240502 21:27:22.988334 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5a87719c2544a5b0cb591ed9f46df8 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:23.009167 27927 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:23.032644 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98b5668e0d504db5a7874f041c14dbfa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:23.034718 26609 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.046787 27963 raft_consensus.cc:1050] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:23.047988 27963 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:23.049829 26742 raft_consensus.cc:1203] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:23.052253 27917 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:23.054729 24520 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:23.059631 26611 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.060775 24520 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.061435 27947 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.061908 24240 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:23.064169 27922 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:23.065142 24782 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.066008 27947 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.066318 27637 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.069221 24390 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.073671 27963 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.079409 26610 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.085384 27969 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.088419 27997 tablet_replica.cc:330] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:23.088656 27997 raft_consensus.cc:2229] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.088832 27997 pending_rounds.cc:62] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Trying to abort 1 pending ops.
I20240502 21:27:23.088923 27997 pending_rounds.cc:68] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350752767180800 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "98b5668e0d504db5a7874f041c14dbfa" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } } }
I20240502 21:27:23.089409 27997 raft_consensus.cc:2894] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:27:23.089519 27997 raft_consensus.cc:2866] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:23.089627 27997 raft_consensus.cc:2258] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.093422 24522 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.096483 26609 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.103909 24782 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.117583 24240 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
W20240502 21:27:23.127249 24653 consensus_meta.cc:287] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Time spent flushing consensus metadata: real 0.523s user 0.004s sys 0.000s
I20240502 21:27:23.128551 24240 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:23.155014 24782 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:23.162678 24650 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
W20240502 21:27:23.173331 27613 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.045s sys 0.132s
W20240502 21:27:23.173858 27613 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.691s user 0.045s sys 0.132s
W20240502 21:27:23.175153 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 222190us
I20240502 21:27:23.177179 27997 ts_tablet_manager.cc:1875] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:23.177778 28002 tablet_replica.cc:330] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:23.178387 28002 raft_consensus.cc:2229] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.178781 28002 raft_consensus.cc:2258] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.180001 28000 raft_consensus.cc:1050] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:23.180857 28000 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:23.184075 28005 raft_consensus.cc:1050] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:23.184350 27997 ts_tablet_manager.cc:1888] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:23.184491 27997 log.cc:1196] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/98b5668e0d504db5a7874f041c14dbfa
I20240502 21:27:23.185334 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 98b5668e0d504db5a7874f041c14dbfa (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:23.208484 24389 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:23.212556 24784 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.212572 24389 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:23.213472 27969 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.216780 27586 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.219571 24653 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:23.220214 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:23.220445 24784 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.222775 27969 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.226197 24653 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.232564 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:23.234081 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b6a4a74e0544e0e99e734a145c91d3b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:23.237810 26610 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:23.238147 24244 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.238646 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:23.240657 26611 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.240993 24242 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 6 to 7, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:23.243858 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6a4a74e0544e0e99e734a145c91d3b with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:23.244379 27997 tablet_replica.cc:330] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:23.244700 27997 raft_consensus.cc:2229] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.245218 27997 raft_consensus.cc:2258] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.246656 27997 ts_tablet_manager.cc:1875] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:23.249100 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8a25580bc74004b0e31c34f841d2cc with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:23.255693 27997 ts_tablet_manager.cc:1888] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:23.256227 24522 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:23.268752 27997 log.cc:1196] T 2b6a4a74e0544e0e99e734a145c91d3b P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b6a4a74e0544e0e99e734a145c91d3b
I20240502 21:27:23.270740 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 2b6a4a74e0544e0e99e734a145c91d3b (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
I20240502 21:27:23.270153 26476 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
W20240502 21:27:23.272514 27853 thread.cc:636] tablet-download-pool-9963dcea478749e0a6e076fac6537fc4 [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.190s sys 0.344s
W20240502 21:27:23.272646 27853 thread.cc:602] tablet-download-pool-9963dcea478749e0a6e076fac6537fc4 [worker] (thread pool) Time spent starting thread: real 0.845s user 0.190s sys 0.344s
I20240502 21:27:23.273242 26742 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:23.274732 27657 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:23.276088 24653 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:23.276715 27657 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:23.284042 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82fb425a5430426f8030ba5497736b11. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:23.287715 27657 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.290817 28002 ts_tablet_manager.cc:1875] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:23.295385 24390 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.298792 26612 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.299933 27969 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.301740 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fc717a1b12e4380a4107e6deae2b837 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:23.303779 26742 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.306264 24389 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:23.306550 27613 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.307111 27613 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:23.308221 27855 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:23.309113 27613 ts_tablet_manager.cc:1367] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.859s user 0.101s sys 0.137s
I20240502 21:27:23.310096 27613 raft_consensus.cc:348] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.310403 27613 raft_consensus.cc:729] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:23.310650 27613 consensus_queue.cc:260] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.311291 27613 ts_tablet_manager.cc:1398] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:23.313464 24389 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
W20240502 21:27:23.328727 27740 consensus_meta.cc:287] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Time spent flushing consensus metadata: real 0.711s user 0.004s sys 0.000s
I20240502 21:27:23.329335 27740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.16.122.2:36251 (request call id 5) took 1298 ms. Trace:
I20240502 21:27:23.329420 24242 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.329516 27740 rpcz_store.cc:276] 0502 21:27:22.030837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:22.030916 (+ 79us) service_pool.cc:225] Handling call
0502 21:27:22.031540 (+ 624us) ts_tablet_manager.cc:903] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
0502 21:27:22.312952 (+281412us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:22.616841 (+303889us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:22.617090 (+ 249us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:27:23.328924 (+711834us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:23.329106 (+ 182us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:23.329226 (+ 120us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:23.329256 (+ 30us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":126,"tablet-copy.queue_time_us":260,"thread_start_us":242419,"threads_started":4}
I20240502 21:27:23.336792 27997 tablet_replica.cc:330] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:23.337266 27997 raft_consensus.cc:2229] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.336841 26612 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:23.337877 27997 raft_consensus.cc:2258] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.339596 27997 ts_tablet_manager.cc:1875] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:23.355284 26477 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.358018 27740 tablet_copy_client.cc:783] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:23.358554 27740 tablet_copy_client.cc:647] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:23.366011 27740 tablet_copy_client.cc:523] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:23.372406 27915 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.16.122.2:36251 (request call id 7) took 1134 ms. Trace:
I20240502 21:27:23.372613 27915 rpcz_store.cc:276] 0502 21:27:22.237651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:22.237714 (+ 63us) service_pool.cc:225] Handling call
0502 21:27:22.284472 (+ 46758us) ts_tablet_manager.cc:903] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
0502 21:27:22.729011 (+444539us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:23.342232 (+613221us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:23.342433 (+ 201us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:27:23.372038 (+ 29605us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:23.372188 (+ 150us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:23.372329 (+ 141us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:23.372357 (+ 28us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":145,"tablet-copy.queue_time_us":44571,"thread_start_us":400054,"threads_started":5}
I20240502 21:27:23.373286 27915 tablet_copy_client.cc:783] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:23.373641 27915 tablet_copy_client.cc:647] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:23.375644 27915 tablet_copy_client.cc:523] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:23.381959 26479 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
W20240502 21:27:23.393947 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.2:43159, user_credentials={real_user=slave}} blocked reactor thread for 217962us
W20240502 21:27:23.395078 27966 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.065s sys 0.119s
I20240502 21:27:23.395714 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:42083)
W20240502 21:27:23.395272 27966 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s user 0.065s sys 0.119s
I20240502 21:27:23.396636 27966 ts_tablet_manager.cc:1398] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.533s user 0.067s sys 0.119s
W20240502 21:27:23.396739 27966 ts_tablet_manager.cc:1430] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Tablet startup took 533ms
I20240502 21:27:23.290761 24650 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.399291 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:42083: session id = 9b0952b24bd8400c9940dbd5e28417b4-7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:23.312314 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-dc5c40e2bf0340958ecd3e7541209e95 received from {username='slave'} at 127.16.122.8:38365
I20240502 21:27:23.402482 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-dc5c40e2bf0340958ecd3e7541209e95 on tablet dc5c40e2bf0340958ecd3e7541209e95 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:23.402781 24804 tablet_copy_source_session.cc:205] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:23.327980 27917 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.403049 27922 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.405479 27915 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:23.405683 28005 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:27:23.396876 27966 ts_tablet_manager.cc:1432] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Trace:
0502 21:27:22.587841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:22.587920 (+ 79us) service_pool.cc:225] Handling call
0502 21:27:22.601956 (+ 14036us) ts_tablet_manager.cc:903] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
0502 21:27:22.658883 (+ 56927us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:22.677086 (+ 18203us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:22.677210 (+ 124us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:27:22.682780 (+ 5570us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:22.682908 (+ 128us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:22.683018 (+ 110us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:22.683042 (+ 24us) inbound_call.cc:173] Queueing success response
0502 21:27:22.708794 (+ 25752us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 21:27:22.863372 (+154578us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 21:27:22.863626 (+ 254us) tablet_replica.cc:259] Starting instrumentation
0502 21:27:22.863911 (+ 285us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":90,"mutex_wait_us":8904,"raft.queue_time_us":44436,"raft.run_cpu_time_us":31,"raft.run_wall_time_us":31,"tablet-copy.queue_time_us":13448,"tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.queue_time_us":41,"tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.run_cpu_time_us":761,"tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.run_wall_time_us":2691,"thread_start_us":579710,"threads_started":6}
I20240502 21:27:23.408623 27740 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:23.425099 24522 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:23.429162 26742 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:23.430111 24650 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:23.431859 27662 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:23.432418 27662 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:23.439306 27855 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.447197 24650 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:23.448086 28019 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:23.454533 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fc717a1b12e4380a4107e6deae2b837 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:23.457407 26478 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:23.458329 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:23.460211 26478 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.465955 27969 raft_consensus.cc:1050] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:23.467075 27969 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:23.470813 24653 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:23.472121 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:23.474830 26476 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:23.475715 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:23.478886 28002 ts_tablet_manager.cc:1888] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:23.479156 28002 log.cc:1196] T 2e5a87719c2544a5b0cb591ed9f46df8 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/2e5a87719c2544a5b0cb591ed9f46df8
I20240502 21:27:23.478986 27586 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.480545 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 2e5a87719c2544a5b0cb591ed9f46df8 (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:23.481634 27997 ts_tablet_manager.cc:1888] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:27:23.481913 27997 log.cc:1196] T 9d8a25580bc74004b0e31c34f841d2cc P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:23.482964 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 9d8a25580bc74004b0e31c34f841d2cc (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:23.485430 24653 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.488667 27922 raft_consensus.cc:1050] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:23.489548 27922 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:23.491010 26476 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.493924 26742 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.494725 26612 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:23.495491 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:23.497284 26743 raft_consensus.cc:1203] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:23.497064 27637 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.497534 26612 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.504514 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:23.507057 24241 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.512945 27953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.514847 24522 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:23.515928 26344 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.516922 24522 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.517988 26344 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.518061 24390 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.518996 24389 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.520226 27953 tablet_copy_client.cc:783] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:23.520512 27953 tablet_copy_client.cc:647] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:23.521484 27853 tablet_copy_client.cc:308] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:23.522310 26742 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.523138 27922 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.527541 24389 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.529242 27851 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.529716 27851 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:23.530015 27851 ts_tablet_manager.cc:1367] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.708s user 0.055s sys 0.003s
I20240502 21:27:23.535424 24522 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.538444 28000 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:23.538556 28012 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.539037 27953 tablet_copy_client.cc:523] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:23.539145 28012 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.539275 27632 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.541347 28000 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.543902 24782 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.544600 27953 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:23.545275 28000 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.511273 24240 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.551366 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:23.551863 24244 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:23.554337 24650 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.558869 28011 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.559634 28011 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.564276 27851 raft_consensus.cc:348] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.564648 27851 raft_consensus.cc:729] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:23.565274 27851 consensus_queue.cc:260] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.567018 27851 ts_tablet_manager.cc:1398] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.037s user 0.004s sys 0.000s
I20240502 21:27:23.567811 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-adde18faca794600bb4aa24f0cf744e6 received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:23.567976 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-adde18faca794600bb4aa24f0cf744e6 on tablet adde18faca794600bb4aa24f0cf744e6 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:23.569363 26476 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.587638 24388 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.588222 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-855b247f654d41c2bbb13abb25d35c1c received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:23.588373 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-855b247f654d41c2bbb13abb25d35c1c on tablet 855b247f654d41c2bbb13abb25d35c1c with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:23.590262 27917 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:23.597251 24718 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8a9b14a7ff04ddab0d1e1a8e9fcb9b0. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:23.597999 24457 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.3:38427, user_credentials={real_user=slave}} blocked reactor thread for 167090us
I20240502 21:27:23.602166 24653 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.607491 26459 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 with delete_type TABLET_DATA_TOMBSTONED (TS 836775e329f148e5908a1afc54f06428 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51488
I20240502 21:27:23.632424 24784 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:23.633656 24247 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:23.641608 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 049bec57d2474c328d2e26e16a4d33a9 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:23.654953 28061 tablet_replica.cc:330] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:23.655690 27739 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.656226 27739 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:23.656159 26742 raft_consensus.cc:1203] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:23.656579 27739 ts_tablet_manager.cc:1367] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 1.685s user 0.065s sys 0.000s
I20240502 21:27:23.656921 28061 raft_consensus.cc:2229] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.657193 28061 pending_rounds.cc:62] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Trying to abort 1 pending ops.
I20240502 21:27:23.657307 28061 pending_rounds.cc:68] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350752741769216 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "049bec57d2474c328d2e26e16a4d33a9" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } } }
I20240502 21:27:23.657913 28061 raft_consensus.cc:2894] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:27:23.658088 28061 raft_consensus.cc:2866] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:23.658296 28061 raft_consensus.cc:2258] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:23.658243 27922 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.658610 27739 raft_consensus.cc:348] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:23.662341 27739 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:23.662696 27739 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:23.666908 27739 ts_tablet_manager.cc:1398] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:27:23.667949 24241 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:23.669880 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-ac35b080037646b4b8b34c0eab2de4e6 received from {username='slave'} at 127.16.122.7:35031
I20240502 21:27:23.670047 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-ac35b080037646b4b8b34c0eab2de4e6 on tablet ac35b080037646b4b8b34c0eab2de4e6 with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:23.674209 28063 ts_tablet_manager.cc:902] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:23.675843 26742 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.688208 28061 ts_tablet_manager.cc:1875] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:23.693018 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 9963dcea478749e0a6e076fac6537fc4 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:23.693277 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 9963dcea478749e0a6e076fac6537fc4 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:23.696513 24542 tablet_copy_source_session.cc:205] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:23.699131 28063 tablet_copy_client.cc:308] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:23.698941 24388 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:23.704849 27750 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.717936 27853 ts_tablet_manager.cc:1875] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:23.723132 27750 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:23.723625 27750 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 1.857s user 0.064s sys 0.000s
I20240502 21:27:23.724681 27750 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.725153 27750 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:23.725451 27750 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.729734 28048 tablet_replica.cc:330] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:23.729923 27853 ts_tablet_manager.cc:1888] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:27:23.730247 27853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.730324 28048 raft_consensus.cc:2229] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:23.730657 27753 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.730883 28048 raft_consensus.cc:2258] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 21:27:23.735255 24233 catalog_manager.cc:4810] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): delete failed for tablet 0fc717a1b12e4380a4107e6deae2b837 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fc717a1b12e4380a4107e6deae2b837 already in progress: copying tablet
I20240502 21:27:23.735347 27750 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 21:27:23.737115 28061 ts_tablet_manager.cc:1888] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:23.739363 27853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.16.122.2:33885 (request call id 7) took 1417 ms. Trace:
I20240502 21:27:23.742611 27753 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:23.743956 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-7bf18ffb96d04d5e9ec8ee9563812ba0 received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:23.744109 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-7bf18ffb96d04d5e9ec8ee9563812ba0 on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:23.744233 27753 ts_tablet_manager.cc:1367] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 1.892s user 0.056s sys 0.011s
I20240502 21:27:23.744488 28061 log.cc:1196] T 049bec57d2474c328d2e26e16a4d33a9 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/049bec57d2474c328d2e26e16a4d33a9
I20240502 21:27:23.740895 28005 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.746785 27753 raft_consensus.cc:348] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:23.749125 27753 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:23.751721 27922 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.751994 27753 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:23.750512 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 049bec57d2474c328d2e26e16a4d33a9 (table rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]) successfully deleted
I20240502 21:27:23.758123 24241 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:23.759938 27753 ts_tablet_manager.cc:1398] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:27:23.763695 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-ac35b080037646b4b8b34c0eab2de4e6 received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:23.763826 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-ac35b080037646b4b8b34c0eab2de4e6 on tablet ac35b080037646b4b8b34c0eab2de4e6 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:23.739495 27853 rpcz_store.cc:276] 0502 21:27:22.322388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:22.322457 (+ 69us) service_pool.cc:225] Handling call
0502 21:27:22.330652 (+ 8195us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:22.334302 (+ 3650us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:22.335247 (+ 945us) ts_tablet_manager.cc:888] Shutdown and deleted data from running replica
0502 21:27:22.335375 (+ 128us) ts_tablet_manager.cc:903] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
0502 21:27:23.717649 (+1382274us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:23.721664 (+ 4015us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:23.729758 (+ 8094us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:23.730114 (+ 356us) tablet_copy_client.cc:448] Replaced tombstoned tablet metadata.
0502 21:27:23.738941 (+ 8827us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:23.739099 (+ 158us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:23.739265 (+ 166us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:23.739294 (+ 29us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":92,"mutex_wait_us":33,"tablet-copy.queue_time_us":71,"thread_start_us":1185647,"threads_started":4}
I20240502 21:27:23.771649 27853 tablet_copy_client.cc:783] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:23.771396 24390 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.773618 27853 tablet_copy_client.cc:647] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:23.780470 24519 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:23.793715 27853 tablet_copy_client.cc:523] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:23.794286 27915 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.794914 27915 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:23.794790 24522 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:23.795070 24520 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.795387 27915 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.390s user 0.058s sys 0.006s
I20240502 21:27:23.796344 27915 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:23.796692 27915 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:23.796969 27915 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:23.797888 27915 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:23.798627 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-9963dcea478749e0a6e076fac6537fc4 received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:23.798772 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-9963dcea478749e0a6e076fac6537fc4 on tablet 9963dcea478749e0a6e076fac6537fc4 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:23.799036 27853 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:23.803725 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 82fb425a5430426f8030ba5497736b11 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:36629)
I20240502 21:27:23.803941 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 82fb425a5430426f8030ba5497736b11 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:36629: session id = b77f8f52372241d483f457389fda0f93-82fb425a5430426f8030ba5497736b11
I20240502 21:27:23.807294 24673 tablet_copy_source_session.cc:205] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:23.808490 26612 raft_consensus.cc:1203] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:23.809176 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb425a5430426f8030ba5497736b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.813072 24388 consensus_queue.cc:237] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:23.817358 28063 tablet_copy_client.cc:783] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:23.818218 28063 tablet_copy_client.cc:647] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:23.820561 28063 tablet_copy_client.cc:523] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:23.824321 28063 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:23.835850 24653 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:23.836777 27969 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:23.838814 27953 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.839272 27953 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:23.839597 27953 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.296s user 0.059s sys 0.002s
I20240502 21:27:23.840418 27953 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:23.840775 27953 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:23.841025 27953 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:23.842499 27953 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:23.843798 27969 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:23.845175 24653 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:23.851305 24247 catalog_manager.cc:5484] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
W20240502 21:27:23.861932 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.2:43159, user_credentials={real_user=slave}} blocked reactor thread for 50991.6us
I20240502 21:27:23.872066 24522 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:23.872939 28011 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.880348 27853 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.880831 27853 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:23.881135 27853 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.084s user 0.049s sys 0.025s
I20240502 21:27:23.881721 26344 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.882102 27853 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:23.882450 27853 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:23.882743 27853 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:23.892226 28063 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:23.892585 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-7a35079a106c4cca94cf65ffa93aefbf received from {username='slave'} at 127.16.122.7:42083
I20240502 21:27:23.892762 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-7a35079a106c4cca94cf65ffa93aefbf on tablet 7a35079a106c4cca94cf65ffa93aefbf with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:23.892812 28063 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:23.893091 28063 ts_tablet_manager.cc:1367] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.069s user 0.044s sys 0.012s
I20240502 21:27:23.893173 26342 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.893950 28063 raft_consensus.cc:348] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:23.894304 28063 raft_consensus.cc:729] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:23.894613 28063 consensus_queue.cc:260] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:23.894332 24522 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:23.913993 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.914817 26612 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.917390 27855 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.917986 27855 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.921856 24781 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:23.925045 27657 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:23.930814 28048 ts_tablet_manager.cc:1875] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:23.932821 26479 raft_consensus.cc:1203] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 21:27:23.948613 24455 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:23.955287 27662 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:23.970829 24784 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:23.981106 27917 raft_consensus.cc:1050] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:23.981984 27917 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:23.993136 24519 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.994122 26742 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:23.994822 27922 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.995378 27922 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:23.909675 24244 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:24.008862 27637 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.012310 24389 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.013897 24519 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.015234 26609 raft_consensus.cc:1203] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:24.019644 26743 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.021384 26742 raft_consensus.cc:1203] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:24.030385 24242 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.030931 27915 ts_tablet_manager.cc:902] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:24.031344 24242 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 5 to 6, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:24.041690 24390 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:24.043864 24387 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.044013 24386 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 21:27:24.044059 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 616e911db4384c17905713adc595ea0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:24.044751 27586 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:24.045390 26479 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:24.045471 26609 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:24.046033 27586 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:24.046855 27969 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:24.047215 24653 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:24.048709 27969 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.051571 27657 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.053414 26479 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.053755 24781 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:24.057556 26342 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.059439 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.060235 26609 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.061348 24242 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:24.066697 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8a25580bc74004b0e31c34f841d2cc with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:24.066761 24522 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.067433 26742 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.068562 27917 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.073027 27922 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.075156 24519 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.077908 27637 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.079491 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc5c40e2bf0340958ecd3e7541209e95 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.082149 24519 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.081796 24244 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:24.085541 26743 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.089198 28048 ts_tablet_manager.cc:1888] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:27:24.091192 28116 tablet_replica.cc:330] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:24.092518 27657 raft_consensus.cc:1050] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:24.094300 27657 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
W20240502 21:27:24.098366 24455 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:24.101567 28116 raft_consensus.cc:2229] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.102026 26479 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.103760 26609 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:24.091719 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5c40e2bf0340958ecd3e7541209e95 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:24.104549 27662 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.104782 28048 log.cc:1196] T 0fc717a1b12e4380a4107e6deae2b837 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:24.105284 28116 raft_consensus.cc:2258] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.107375 24783 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:24.108387 28005 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.107223 26609 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.111680 28005 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240502 21:27:24.112782 27915 tablet_copy_client.cc:308] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:24.112691 28019 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.114096 28042 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:24.117223 26342 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.124562 24522 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.124598 24241 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.127458 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:24.128393 27662 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:24.142659 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff7e019fa82d4648afaf130d3f6343d6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.156071 27657 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.160007 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7e019fa82d4648afaf130d3f6343d6 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:24.159854 24782 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.161538 26479 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.165508 24784 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:24.168284 24241 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 6 to 7, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.170706 24783 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.178071 26342 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.178874 28019 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.182265 28000 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.187762 28042 raft_consensus.cc:1050] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:24.188783 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.188815 28042 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
W20240502 21:27:24.192077 24455 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:24.193022 24783 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:24.193034 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:24.191066 26342 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.193794 27855 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:24.196019 27657 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:24.199826 27586 raft_consensus.cc:1050] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:24.201364 27586 consensus_queue.cc:237] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:24.207227 26476 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.216640 24240 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.219370 24518 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.206182 28119 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.222638 28019 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.224421 24240 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:24.224500 24518 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:24.228765 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:24.237073 24520 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:24.238368 28116 ts_tablet_manager.cc:1875] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:24.239161 24388 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.246894 24518 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.248484 24784 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.190800 26609 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.254874 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:24.256043 28119 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 21:27:24.265867 24232 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.266772 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 0fc717a1b12e4380a4107e6deae2b837 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:24.267484 28048 tablet_replica.cc:330] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:24.267828 28048 raft_consensus.cc:2229] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.268261 28048 raft_consensus.cc:2258] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:24.269301 28048 ts_tablet_manager.cc:1875] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:24.311533 27657 raft_consensus.cc:1050] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:24.312588 27657 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:24.313477 27969 raft_consensus.cc:1050] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:24.314450 27969 consensus_queue.cc:237] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.319478 24232 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 7: aborting the task: latest config opid_index 8; task opid_index 7
I20240502 21:27:24.326560 28137 ts_tablet_manager.cc:902] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:24.330168 24650 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.331324 27963 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:24.337435 26743 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.338579 27963 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:24.346033 24783 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:24.347152 28131 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:24.348937 28151 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.350874 28131 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.357013 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.358721 26476 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.368738 24240 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.382992 26742 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
W20240502 21:27:24.385773 26265 kernel_stack_watchdog.cc:198] Thread 27853 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 323ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f79bad53980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f79b79c1d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f79b79e27ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f79b79de3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f79b79dc84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f79c31d64f0 kudu::consensus::RaftConsensus::MarkDirty() at ??:0
@ 0x7f79c31cf1b0 kudu::consensus::RaftConsensus::Start() at ??:0
@ 0x7f79c44e469e kudu::tablet::TabletReplica::Start() at ??:0
@ 0x7f79c6f75554 kudu::tserver::TSTabletManager::OpenTablet() at ??:0
@ 0x7f79c6f70a29 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f79c6fa8a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f79c6f8b981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f79c6f8b70e std::_Function_handler<>::_M_invoke() at ??:0
I20240502 21:27:24.386660 24783 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.387892 24519 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.389405 28156 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 21:27:24.391352 24240 catalog_manager.cc:5484] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.395866 26325 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 with delete_type TABLET_DATA_TOMBSTONED (TS b77f8f52372241d483f457389fda0f93 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38914
I20240502 21:27:24.395848 24782 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.398483 26610 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:24.399056 26611 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:24.399904 26612 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:24.399982 28131 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:24.405297 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:24.405465 26612 raft_consensus.cc:858] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
W20240502 21:27:24.405827 26612 raft_consensus.cc:1565] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.405961 28119 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:24.407011 26612 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:24.412678 24783 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.421342 26476 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.422128 28131 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:27:24.423231 27853 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.056s sys 0.095s
W20240502 21:27:24.423395 27853 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.056s sys 0.095s
W20240502 21:27:24.423866 28063 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.068s sys 0.112s
W20240502 21:27:24.424147 28063 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.068s sys 0.112s
I20240502 21:27:24.424580 28063 ts_tablet_manager.cc:1398] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.531s user 0.070s sys 0.112s
W20240502 21:27:24.424696 28063 ts_tablet_manager.cc:1430] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Tablet startup took 531ms
W20240502 21:27:24.424778 28063 ts_tablet_manager.cc:1432] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Trace:
0502 21:27:23.668912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:23.668976 (+ 64us) service_pool.cc:225] Handling call
0502 21:27:23.674329 (+ 5353us) ts_tablet_manager.cc:903] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
0502 21:27:23.808606 (+134277us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:23.812531 (+ 3925us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:23.812638 (+ 107us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:27:23.815685 (+ 3047us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:23.816012 (+ 327us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:23.816148 (+ 136us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:23.816175 (+ 27us) inbound_call.cc:173] Queueing success response
0502 21:27:23.824094 (+ 7919us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 21:27:23.893176 (+ 69082us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 21:27:23.893381 (+ 205us) tablet_replica.cc:259] Starting instrumentation
0502 21:27:23.893540 (+ 159us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":93,"raft.queue_time_us":5672,"raft.run_cpu_time_us":29,"raft.run_wall_time_us":29,"spinlock_wait_cycles":14464,"tablet-copy.queue_time_us":4961,"tablet-download-pool-82fb425a5430426f8030ba5497736b11.queue_time_us":52,"tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_cpu_time_us":618,"tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_wall_time_us":1890,"thread_start_us":557075,"threads_started":6}
I20240502 21:27:24.425626 26611 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.426278 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-82fb425a5430426f8030ba5497736b11 received from {username='slave'} at 127.16.122.5:36629
I20240502 21:27:24.426448 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session b77f8f52372241d483f457389fda0f93-82fb425a5430426f8030ba5497736b11 on tablet 82fb425a5430426f8030ba5497736b11 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:24.427526 24650 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.428493 24780 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.340304 28137 tablet_copy_client.cc:308] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:24.430794 24784 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.431326 24650 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.431393 28131 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.432216 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 616e911db4384c17905713adc595ea0a from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:24.432293 28119 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.432422 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 616e911db4384c17905713adc595ea0a from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-616e911db4384c17905713adc595ea0a
I20240502 21:27:24.432628 26342 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:24.433211 28119 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:24.433256 27586 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:24.434567 28150 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 21:27:24.436594 24410 tablet_copy_source_session.cc:205] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:24.443723 28156 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:24.454007 27853 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.573s user 0.059s sys 0.096s
W20240502 21:27:24.454145 27853 ts_tablet_manager.cc:1430] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Tablet startup took 572ms
W20240502 21:27:24.454217 27853 ts_tablet_manager.cc:1432] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Trace:
0502 21:27:22.322388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:22.322457 (+ 69us) service_pool.cc:225] Handling call
0502 21:27:22.330652 (+ 8195us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:22.334302 (+ 3650us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:22.335247 (+ 945us) ts_tablet_manager.cc:888] Shutdown and deleted data from running replica
0502 21:27:22.335375 (+ 128us) ts_tablet_manager.cc:903] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
0502 21:27:23.717649 (+1382274us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:27:23.721664 (+ 4015us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:23.729758 (+ 8094us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:23.730114 (+ 356us) tablet_copy_client.cc:448] Replaced tombstoned tablet metadata.
0502 21:27:23.738941 (+ 8827us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:27:23.739099 (+ 158us) tablet_replica.cc:205] Creating consensus instance
0502 21:27:23.739265 (+ 166us) ts_tablet_manager.cc:958] Replica registered.
0502 21:27:23.739294 (+ 29us) inbound_call.cc:173] Queueing success response
0502 21:27:23.797382 (+ 58088us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 21:27:23.881222 (+ 83840us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 21:27:23.881506 (+ 284us) tablet_replica.cc:259] Starting instrumentation
0502 21:27:23.881639 (+ 133us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":92,"mutex_wait_us":585,"raft.queue_time_us":17224,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":65,"spinlock_wait_cycles":7424,"tablet-copy.queue_time_us":71,"tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us":34,"tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_cpu_time_us":672,"tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_wall_time_us":19809,"thread_start_us":1725971,"threads_started":5}
I20240502 21:27:24.455485 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-9963dcea478749e0a6e076fac6537fc4 received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:24.455641 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-9963dcea478749e0a6e076fac6537fc4 on tablet 9963dcea478749e0a6e076fac6537fc4 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:24.458122 28042 raft_consensus.cc:1013] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: attempt to promote peer b77f8f52372241d483f457389fda0f93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:27:24.458426 24390 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:24.459126 24384 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:24.459652 24385 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 21:27:24.466987 24232 catalog_manager.cc:4797] TS b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): delete failed for tablet ac35b080037646b4b8b34c0eab2de4e6 because the tablet was not found. No further retry: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:24.468171 27657 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.470829 24780 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
W20240502 21:27:24.483700 28141 log.cc:924] Time spent T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240502 21:27:24.490810 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e911db4384c17905713adc595ea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.490525 28150 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.493747 28116 ts_tablet_manager.cc:1888] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:24.493927 28116 log.cc:1196] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:24.519179 28156 raft_consensus.cc:1050] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:24.520148 28156 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:24.530107 28137 tablet_copy_client.cc:783] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:24.535218 26612 raft_consensus.cc:858] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
W20240502 21:27:24.535377 26612 raft_consensus.cc:1565] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.546788 26612 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.552245 28137 tablet_copy_client.cc:647] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:24.552403 24781 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:24.557488 28131 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:24.557049 26611 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.558687 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34643)
I20240502 21:27:24.558910 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34643: session id = 836775e329f148e5908a1afc54f06428-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0
I20240502 21:27:24.559141 28137 tablet_copy_client.cc:523] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:24.563555 24247 catalog_manager.cc:5484] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.565789 28119 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.570922 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.572301 24804 tablet_copy_source_session.cc:205] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:24.573904 27915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:24.573437 24781 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.574193 26742 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.574009 26476 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.575625 28157 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:24.577153 24247 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 9 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:24.581467 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:24.582793 24781 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:24.583129 24520 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:24.583376 28156 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:24.585897 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:24.586697 27915 tablet_copy_client.cc:783] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:24.586829 28042 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:24.586910 27915 tablet_copy_client.cc:647] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:24.588872 24782 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.590044 27915 tablet_copy_client.cc:523] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:24.590652 28042 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.594022 27915 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:24.595191 24240 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.596724 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:24.841063 24502 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:24.615574 28137 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:24.635051 24652 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.617532 24246 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 10 to 11, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:24.968912 24501 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:24.613996 28156 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.611049 28000 raft_consensus.cc:1050] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:24.597556 26476 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:24.606370 26611 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:24.605530 26342 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.636850 24653 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:25.312152 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:24.611897 24781 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:24.616102 26743 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:24.640918 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 9d8a25580bc74004b0e31c34f841d2cc (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
W20240502 21:27:25.330587 24234 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.122.3:38427, user_credentials={real_user=slave}} blocked reactor thread for 689560us
I20240502 21:27:24.605232 26479 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.622746 28048 ts_tablet_manager.cc:1888] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:24.632736 24389 consensus_queue.cc:237] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:25.331247 28048 log.cc:1196] T dc5c40e2bf0340958ecd3e7541209e95 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc5c40e2bf0340958ecd3e7541209e95
I20240502 21:27:25.332124 28048 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 27) took 1240 ms. Trace:
I20240502 21:27:25.271581 28157 raft_consensus.cc:482] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1398472b92c842f285b8c7cc43cc1527)
I20240502 21:27:24.637291 24649 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:25.332530 28157 raft_consensus.cc:504] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:25.332291 28048 rpcz_store.cc:276] 0502 21:27:24.091551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.091626 (+ 75us) service_pool.cc:225] Handling call
0502 21:27:24.267215 (+175589us) ts_tablet_manager.cc:1120] Deleting tablet dc5c40e2bf0340958ecd3e7541209e95
0502 21:27:24.267229 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 21:27:24.485271 (+218042us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:24.622590 (+137319us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:25.332066 (+709476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":162622}
I20240502 21:27:25.333606 28048 tablet_replica.cc:330] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:24.640189 26344 raft_consensus.cc:1203] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:25.333719 28157 leader_election.cc:288] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:27:25.121060 24782 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:25.337338 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet dc5c40e2bf0340958ecd3e7541209e95 (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
W20240502 21:27:25.063884 24221 kernel_stack_watchdog.cc:198] Thread 24234 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 236ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f0e900f3980 (unknown) at ??:0
@ 0x7f0e900ed9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f0e8a073904 google::LogMessage::Flush() at ??:0
@ 0x7f0e8a073a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f0ea02131e8 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7f0ea016bb09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f0ea0216e30 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f0ea0216c4e _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0e939b38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f0e939b34ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0e9399f4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f0e9399fa2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f0e9393c51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f0e9393af45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240502 21:27:25.340762 28048 raft_consensus.cc:2229] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:24.798617 27915 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:25.341521 27915 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:25.341846 27915 ts_tablet_manager.cc:1367] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.748s user 0.054s sys 0.009s
I20240502 21:27:25.341099 28048 raft_consensus.cc:2258] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.342813 28048 ts_tablet_manager.cc:1875] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:25.342851 27915 raft_consensus.cc:348] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:25.343098 27915 raft_consensus.cc:729] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:25.343446 27915 consensus_queue.cc:260] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:25.343536 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
I20240502 21:27:24.638052 24648 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:24.828539 27740 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:25.348786 27740 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:25.349123 27740 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 1.941s user 0.070s sys 0.023s
I20240502 21:27:25.352227 24779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
I20240502 21:27:25.354223 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:25.354620 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.358943 28276 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:25.363875 26611 raft_consensus.cc:858] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
W20240502 21:27:25.364080 26611 raft_consensus.cc:1565] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:25.364708 26611 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:25.367372 28153 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:25.368876 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.373090 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:25.350178 27740 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:25.377099 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:25.377103 27740 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:25.377422 27740 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:25.378571 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.403928 28048 ts_tablet_manager.cc:1888] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:25.404198 28048 log.cc:1196] T ff7e019fa82d4648afaf130d3f6343d6 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff7e019fa82d4648afaf130d3f6343d6
I20240502 21:27:25.420395 24651 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.421286 26723 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.421700 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.423825 28156 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:25.425529 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.426378 27740 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.077s user 0.004s sys 0.000s
I20240502 21:27:25.427301 24542 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-0fc717a1b12e4380a4107e6deae2b837 received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:25.427493 24542 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-0fc717a1b12e4380a4107e6deae2b837 on tablet 0fc717a1b12e4380a4107e6deae2b837 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:25.438939 28150 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:25.439687 24651 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:25.446058 28048 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 28) took 1286 ms. Trace:
I20240502 21:27:25.452083 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adde18faca794600bb4aa24f0cf744e6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:25.446445 28048 rpcz_store.cc:276] 0502 21:27:24.159819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.159884 (+ 65us) service_pool.cc:225] Handling call
0502 21:27:25.333446 (+1173562us) ts_tablet_manager.cc:1120] Deleting tablet ff7e019fa82d4648afaf130d3f6343d6
0502 21:27:25.333459 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 21:27:25.371154 (+ 37695us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:25.403669 (+ 32515us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:25.445877 (+ 42208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6895,"tablet-delete.queue_time_us":1173278}
I20240502 21:27:25.453248 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet ff7e019fa82d4648afaf130d3f6343d6 (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
I20240502 21:27:25.480860 28048 tablet_replica.cc:330] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:25.481173 28048 raft_consensus.cc:2229] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:25.481669 28048 raft_consensus.cc:2258] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.482498 28048 ts_tablet_manager.cc:1875] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:25.469870 24653 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
W20240502 21:27:25.503846 28000 raft_consensus.cc:1054] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Unable to promote non-voter 9b0952b24bd8400c9940dbd5e28417b4: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:25.505102 26611 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:25.508298 24762 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:25.508780 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:25.511755 26743 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.514037 28150 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:25.517238 28048 ts_tablet_manager.cc:1888] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:25.517375 28048 log.cc:1196] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/7b2661e0b5fe4a0b94c150b14b8fd9ec
I20240502 21:27:25.518000 28048 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 29) took 1289 ms. Trace:
I20240502 21:27:25.518100 28048 rpcz_store.cc:276] 0502 21:27:24.228602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.228663 (+ 61us) service_pool.cc:225] Handling call
0502 21:27:25.455533 (+1226870us) ts_tablet_manager.cc:1120] Deleting tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec
0502 21:27:25.455547 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 21:27:25.486241 (+ 30694us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:25.517073 (+ 30832us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:25.517925 (+ 852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226593}
I20240502 21:27:25.518621 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec (table rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]) successfully deleted
I20240502 21:27:25.519090 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:25.540494 24322 leader_election.cc:302] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e856847c3f494e6aa57f9cecf117ce4d; no voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832
W20240502 21:27:25.540949 24322 raft_consensus.cc:2670] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P e856847c3f494e6aa57f9cecf117ce4d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 21:27:25.544742 26743 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:25.559458 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.563789 26610 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:25.564513 28019 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:25.575786 26325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.575911 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.577417 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.577672 28276 raft_consensus.cc:1050] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:25.578886 28276 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:25.583439 24782 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:25.583437 26345 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:25.584056 28277 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:25.584659 28277 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:25.589094 24648 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:25.589756 28019 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:25.562220 27915 ts_tablet_manager.cc:1398] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.220s user 0.005s sys 0.007s
I20240502 21:27:25.602180 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 received from {username='slave'} at 127.16.122.6:34643
I20240502 21:27:25.602391 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 836775e329f148e5908a1afc54f06428-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 on tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:25.570698 26459 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.582285 26479 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:25.595309 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.609755 28276 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:25.614837 24651 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:25.615692 28276 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:25.618983 28277 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:25.619906 24649 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:25.619952 26479 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:25.626921 26344 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:25.628103 28019 raft_consensus.cc:1013] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: attempt to promote peer 9b0952b24bd8400c9940dbd5e28417b4: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:27:25.629483 28210 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.634673 28210 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44460 (request call id 1) took 1024 ms. Trace:
I20240502 21:27:25.634847 28210 rpcz_store.cc:276] 0502 21:27:24.610491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.610872 (+ 381us) service_pool.cc:225] Handling call
0502 21:27:25.360809 (+749937us) log.cc:842] Serialized 618 byte log entry
0502 21:27:25.634622 (+273813us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":97,"raft.queue_time_us":56660,"raft.run_cpu_time_us":936,"raft.run_wall_time_us":936,"thread_start_us":140871,"threads_started":1,"wal-append.queue_time_us":139538}
I20240502 21:27:25.646179 24649 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.655496 26610 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.656067 28048 tablet_replica.cc:330] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:25.656308 28048 raft_consensus.cc:2229] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:25.656673 28048 raft_consensus.cc:2258] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.664917 24520 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:25.670078 24780 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:25.673626 28210 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:25.684147 24520 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.688495 24246 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.16.122.2:51399 (request call id 188) took 1080 ms. Trace:
I20240502 21:27:25.690786 26458 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:51488
I20240502 21:27:25.688697 24246 rpcz_store.cc:276] 0502 21:27:24.607890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.607938 (+ 48us) service_pool.cc:225] Handling call
0502 21:27:25.688431 (+1080493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 21:27:25.320708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 21:27:25.320852 (+ 144us) write_op.cc:424] Acquiring schema lock in shared mode
0502 21:27:25.320860 (+ 8us) write_op.cc:427] Acquired schema lock
0502 21:27:25.320862 (+ 2us) tablet.cc:660] Decoding operations
0502 21:27:25.321004 (+ 142us) write_op.cc:612] Acquiring the partition lock for write op
0502 21:27:25.321021 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0502 21:27:25.321028 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0502 21:27:25.321119 (+ 91us) tablet.cc:699] Row locks acquired
0502 21:27:25.321122 (+ 3us) write_op.cc:260] PREPARE: finished
0502 21:27:25.321176 (+ 54us) write_op.cc:270] Start()
0502 21:27:25.321220 (+ 44us) write_op.cc:276] Timestamp: P: 1714685245321113 usec, L: 0
0502 21:27:25.321227 (+ 7us) op_driver.cc:331] REPLICATION: starting
0502 21:27:25.321526 (+ 299us) log.cc:842] Serialized 539 byte log entry
0502 21:27:25.664529 (+343003us) op_driver.cc:444] REPLICATION: finished
0502 21:27:25.677366 (+ 12837us) write_op.cc:301] APPLY: starting
0502 21:27:25.677392 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0502 21:27:25.677440 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0502 21:27:25.677471 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 21:27:25.677554 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 21:27:25.677591 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 21:27:25.677600 (+ 9us) write_op.cc:312] APPLY: finished
0502 21:27:25.677722 (+ 122us) log.cc:842] Serialized 22 byte log entry
0502 21:27:25.677861 (+ 139us) write_op.cc:481] Releasing partition, row and schema locks
0502 21:27:25.677889 (+ 28us) write_op.cc:446] Released schema lock
0502 21:27:25.677899 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":12720,"num_ops":1,"prepare.queue_time_us":471,"prepare.run_cpu_time_us":971,"prepare.run_wall_time_us":8223,"replication_time_us":343261,"thread_start_us":44121,"threads_started":2}]]}
I20240502 21:27:25.691306 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:25.691540 24243 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:25.705787 28000 raft_consensus.cc:1050] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:25.706800 28000 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:25.707504 28048 ts_tablet_manager.cc:1875] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:25.711205 26609 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.711814 28211 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:25.717051 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:25.717916 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:25.720001 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:25.729425 28317 tablet_replica.cc:330] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:25.729689 28317 raft_consensus.cc:2229] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:25.730274 28317 raft_consensus.cc:2258] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:25.743729 28317 ts_tablet_manager.cc:1875] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:25.746562 24247 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.16.122.7:58031 (request call id 36) took 1139 ms. Trace:
I20240502 21:27:25.746728 24247 rpcz_store.cc:276] 0502 21:27:24.607334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.610542 (+ 3208us) service_pool.cc:225] Handling call
0502 21:27:25.746503 (+1135961us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240502 21:27:25.746780 24389 consensus_queue.cc:237] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:25.752553 24242 catalog_manager.cc:5484] T adde18faca794600bb4aa24f0cf744e6 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:25.757153 24239 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.758074 24520 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.757920 24239 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:25.760676 26610 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.761371 28157 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:25.761669 24648 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.762250 28157 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:25.770978 28157 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:25.775025 28210 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:25.775980 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 855b247f654d41c2bbb13abb25d35c1c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 21:27:25.858551 28137 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.083s sys 0.123s
W20240502 21:27:26.314159 28137 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.980s user 0.083s sys 0.123s
W20240502 21:27:26.314203 24233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.122.1:36473, user_credentials={real_user=slave}} blocked reactor thread for 538262us
W20240502 21:27:25.852595 24322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 100857us
I20240502 21:27:26.243947 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:25.776786 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet adde18faca794600bb4aa24f0cf744e6 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:25.777597 26610 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
W20240502 21:27:26.196388 24221 kernel_stack_watchdog.cc:198] Thread 24233 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 246ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f0e900f3980 (unknown) at ??:0
@ 0x7f0e88f15255 __write at ??:0
@ 0x7f0e88e9028d _IO_file_write at ??:0
@ 0x7f0e88e90bff _IO_file_xsputn at ??:0
@ 0x7f0e88e84a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7f0e8a0778a0 google::LogMessage::SendToLog() at ??:0
@ 0x7f0e8a0737cc google::LogMessage::Flush() at ??:0
@ 0x7f0e8a073a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f0ea016ebb1 kudu::master::AsyncChangeConfigTask::HandleResponse() at ??:0
@ 0x7f0ea016bb09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f0ea01aa4b0 kudu::master::AsyncEvictReplicaTask::SendRequest()::$_33::operator()() at ??:0
@ 0x7f0ea01aa2d9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0e939b38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f0e939b34ae std::_Function_handler<>::_M_invoke() at ??:0
I20240502 21:27:25.865891 28048 ts_tablet_manager.cc:1888] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:27:26.316358 28048 log.cc:1196] T ac35b080037646b4b8b34c0eab2de4e6 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:26.316958 28048 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 30) took 1735 ms. Trace:
I20240502 21:27:26.317054 28048 rpcz_store.cc:276] 0502 21:27:24.581287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:24.581341 (+ 54us) service_pool.cc:225] Handling call
0502 21:27:25.655879 (+1074538us) ts_tablet_manager.cc:1120] Deleting tablet ac35b080037646b4b8b34c0eab2de4e6
0502 21:27:25.655892 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 21:27:25.810729 (+154837us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:25.847044 (+ 36315us) tablet_metadata.cc:641] Metadata flushed
0502 21:27:26.316896 (+469852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074224}
I20240502 21:27:25.795908 24520 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:25.796582 24651 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:25.824577 28316 tablet_replica.cc:330] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: stopping tablet replica
I20240502 21:27:26.311439 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:25.804104 28317 ts_tablet_manager.cc:1888] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 21:27:26.322012 28317 log.cc:1196] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:26.264459 26611 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.839129 24519 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:26.322994 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2661e0b5fe4a0b94c150b14b8fd9ec" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b77f8f52372241d483f457389fda0f93" is_pre_election: true
I20240502 21:27:25.789220 24649 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:25.822742 28000 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:25.930557 28318 raft_consensus.cc:482] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e856847c3f494e6aa57f9cecf117ce4d)
W20240502 21:27:25.863704 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a35079a106c4cca94cf65ffa93aefbf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:26.327361 26479 raft_consensus.cc:1203] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 21:27:26.327474 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 463796us
I20240502 21:27:25.889221 28323 raft_consensus.cc:1050] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
W20240502 21:27:26.149513 24706 kernel_stack_watchdog.cc:198] Thread 24717 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 213ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe8603c5980 (unknown) at ??:0
@ 0x7fe8603bf9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fe85a345904 google::LogMessage::Flush() at ??:0
@ 0x7fe85a345a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe8687829ee kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7fe868781adf kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7fe868788061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7fe868787e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe863c858cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fe863c854ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe863c714e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fe863c71a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fe863c0e51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fe863c0cf45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240502 21:27:26.329172 28323 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:26.332082 28211 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:26.333456 26478 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:26.333842 28316 raft_consensus.cc:2229] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 21:27:26.334218 28316 raft_consensus.cc:2258] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 21:27:26.335002 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet ac35b080037646b4b8b34c0eab2de4e6 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:26.335856 28316 ts_tablet_manager.cc:1875] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:26.327474 28318 raft_consensus.cc:504] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:26.339711 28318 leader_election.cc:288] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045)
I20240502 21:27:26.340423 24388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:26.344174 28277 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:26.347208 24648 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:26.347899 24653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:26.348960 28317 tablet_replica.cc:330] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:26.349228 28317 raft_consensus.cc:2229] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:26.349499 28317 raft_consensus.cc:2258] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.350334 28317 ts_tablet_manager.cc:1875] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:26.351305 24240 catalog_manager.cc:5484] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:26.356971 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:26.359375 24719 leader_election.cc:302] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 427345e9a65d417c84a773ac1925a18a, e856847c3f494e6aa57f9cecf117ce4d
W20240502 21:27:26.359675 24719 raft_consensus.cc:2670] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 21:27:26.360935 26344 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:26.363746 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:26.366163 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet ac35b080037646b4b8b34c0eab2de4e6 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:26.369361 28277 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:26.381590 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:26.381851 28277 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.396342 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:26.397334 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:26.400031 26344 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.434145 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9963dcea478749e0a6e076fac6537fc4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:26.439688 26342 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:26.440443 24522 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:26.440778 28316 ts_tablet_manager.cc:1888] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:26.440939 27637 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:26.444748 27637 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:26.445089 24781 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:26.446091 28308 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:26.448185 24781 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.477455 28317 ts_tablet_manager.cc:1888] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:26.477718 28317 log.cc:1196] T adde18faca794600bb4aa24f0cf744e6 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/adde18faca794600bb4aa24f0cf744e6
I20240502 21:27:26.478891 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet adde18faca794600bb4aa24f0cf744e6 (table rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]) successfully deleted
I20240502 21:27:26.500452 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:26.440958 28316 log.cc:1196] T 0fc717a1b12e4380a4107e6deae2b837 P 836775e329f148e5908a1afc54f06428: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/wals/0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:26.444068 26476 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.503203 24233 catalog_manager.cc:4830] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): tablet 0fc717a1b12e4380a4107e6deae2b837 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:26.509900 28000 raft_consensus.cc:1050] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:26.511147 28000 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:26.516571 24241 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
W20240502 21:27:26.526448 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 197895us
I20240502 21:27:26.530175 27637 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:26.530761 26741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adde18faca794600bb4aa24f0cf744e6" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1769ee000eb24298a367a0de1209e6e0" is_pre_election: true
I20240502 21:27:26.536753 24520 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:26.538189 26476 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:26.538863 28210 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:26.539130 24648 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:26.539927 28210 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:26.542083 26342 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:26.549829 24781 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:26.553615 24519 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:26.554683 28323 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:26.587488 28000 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.591217 24649 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.596380 26479 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.612433 24649 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:26.617019 28386 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:26.627502 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:26.627635 24388 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:26.628242 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:26.629676 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:26.642217 24519 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:26.660144 26743 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.659528 28323 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.678777 24384 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.679280 28386 raft_consensus.cc:482] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e856847c3f494e6aa57f9cecf117ce4d)
I20240502 21:27:26.679535 28386 raft_consensus.cc:504] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:26.680999 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:26.683987 28386 leader_election.cc:288] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
I20240502 21:27:26.682240 24781 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:26.685170 26610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4" is_pre_election: true
I20240502 21:27:26.686378 24718 leader_election.cc:302] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 427345e9a65d417c84a773ac1925a18a, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:26.688634 24384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855b247f654d41c2bbb13abb25d35c1c" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:27:26.688385 28386 raft_consensus.cc:2726] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:26.698877 24780 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:26.709587 24242 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.738672 24244 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:26.741257 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 855b247f654d41c2bbb13abb25d35c1c with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:26.741770 28317 tablet_replica.cc:330] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:26.741974 28317 raft_consensus.cc:2229] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:26.742229 28317 raft_consensus.cc:2258] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.742445 28323 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20240502 21:27:26.743083 28317 ts_tablet_manager.cc:1875] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:26.743322 24384 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:26.748097 24519 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:26.755172 28317 ts_tablet_manager.cc:1888] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:26.755331 28317 log.cc:1196] T 855b247f654d41c2bbb13abb25d35c1c P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/855b247f654d41c2bbb13abb25d35c1c
I20240502 21:27:26.756229 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 9963dcea478749e0a6e076fac6537fc4 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:26.756681 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 855b247f654d41c2bbb13abb25d35c1c (table rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]) successfully deleted
I20240502 21:27:26.762926 24780 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.768390 26743 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:26.769615 28157 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:27:26.771489 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53c1992a21224bac883da474f143c1d7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:26.771492 24388 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:26.772430 28383 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:26.777410 24781 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:26.781208 28383 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:26.795596 24649 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:26.796386 28157 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:26.806365 28277 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:26.807271 24781 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:26.837414 28428 tablet_replica.cc:330] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:26.837626 28428 raft_consensus.cc:2229] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:26.837985 28428 raft_consensus.cc:2258] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:26.838721 28428 ts_tablet_manager.cc:1875] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:26.844305 24241 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 4 to 5, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:27.141101 24519 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:26.868814 26743 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.880707 28438 ts_tablet_manager.cc:902] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:26.931622 28137 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:27.158421 28137 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:27.158869 28137 ts_tablet_manager.cc:1367] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 2.543s user 0.145s sys 0.128s
I20240502 21:27:27.159904 28137 raft_consensus.cc:348] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:27.160174 28137 raft_consensus.cc:729] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:27.160401 28137 consensus_queue.cc:260] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:27.163878 28137 ts_tablet_manager.cc:1398] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:27.176079 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:27.218420 28438 tablet_copy_client.cc:308] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:27.187832 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:26.867280 24649 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.198560 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:27.179124 24456 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4aa176fd994cf0b55f304cf9d06fe0. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:27.308728 24456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.6:41751, user_credentials={real_user=slave}} blocked reactor thread for 129553us
I20240502 21:27:26.874914 28428 ts_tablet_manager.cc:1888] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:27:27.309031 28428 log.cc:1196] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:27.192411 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:26.849474 24781 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:27.178030 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:26.844362 28157 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:26.865203 24388 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:26.847730 24247 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.077330 24648 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:27.167822 24651 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:27.173871 24242 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.315847 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.316138 24781 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:26.968073 24384 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
W20240502 21:27:26.903069 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:27.149684 24311 kernel_stack_watchdog.cc:198] Thread 24322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 181ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fdb8e49b980 (unknown) at ??:0
@ 0x7fdb8e4959c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fdb8841b904 google::LogMessage::Flush() at ??:0
@ 0x7fdb8841ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fdb968589ee kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7fdb96857adf kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7fdb9685e061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7fdb9685de6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fdb91d5b8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fdb91d5b4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fdb91d474e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fdb91d47a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fdb91ce451a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fdb91ce2f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240502 21:27:27.166745 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-616e911db4384c17905713adc595ea0a received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:27.317332 24244 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.319501 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-616e911db4384c17905713adc595ea0a on tablet 616e911db4384c17905713adc595ea0a with peer 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:27.320510 24322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 417035us
I20240502 21:27:27.322605 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 9963dcea478749e0a6e076fac6537fc4 (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:27.323606 24522 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.327235 24652 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.328105 26610 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.330781 28308 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:27.335243 26743 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.335920 28323 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.337752 28386 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.339167 24780 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.340158 28323 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.340392 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:27.345219 28386 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.348245 28210 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.351109 26479 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.351631 28528 tablet_replica.cc:330] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:27.353302 24782 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:27.354000 24648 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:27.358868 24522 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:27.362309 26344 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 21:27:27.366477 24234 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:27.367957 28277 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:27.369855 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:27.373407 24519 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:27.373284 28308 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.375180 26479 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:27.375074 28386 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.376576 24653 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:27.378563 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9963dcea478749e0a6e076fac6537fc4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.379599 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:33001)
I20240502 21:27:27.379789 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:33001: session id = 1769ee000eb24298a367a0de1209e6e0-7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:27.379920 28210 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:27.380884 24247 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.383877 24762 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:27.383839 26345 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.384500 26344 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:27.387107 28329 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:27.388643 24519 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.391110 24762 tablet_service.cc:1514] Processing DeleteTablet for tablet 9963dcea478749e0a6e076fac6537fc4 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:27.393323 26479 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:27.394263 28210 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:27.396411 28210 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.399078 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a35079a106c4cca94cf65ffa93aefbf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.401733 28308 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:27.403470 28323 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:27.403600 26476 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.404278 24649 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.409631 26610 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.351882 28528 raft_consensus.cc:2229] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.415936 24244 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.421016 28329 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:27.425922 26344 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:27.431891 24522 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:27.439455 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82fb425a5430426f8030ba5497736b11 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.443337 24244 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:27.451804 24653 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.467463 28528 raft_consensus.cc:2258] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.470016 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf with delete_type TABLET_DATA_TOMBSTONED (TS 1769ee000eb24298a367a0de1209e6e0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:27.483711 28528 ts_tablet_manager.cc:1875] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:27.485029 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fb425a5430426f8030ba5497736b11 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:27.487473 28527 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:27.490640 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.503019 26479 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:27.504238 24247 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:27.507484 24782 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:27.507900 24243 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.518155 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
W20240502 21:27:27.526754 24232 catalog_manager.cc:4797] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): delete failed for tablet 7a35079a106c4cca94cf65ffa93aefbf because the tablet was not found. No further retry: Not found: Tablet not found: 7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:27.537546 24804 tablet_copy_source_session.cc:205] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:27.539855 28438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.547133 28438 tablet_copy_client.cc:783] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:27.547412 28438 tablet_copy_client.cc:647] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:27.549955 28438 tablet_copy_client.cc:523] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:27.508917 24779 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:27.553504 28438 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:27.559818 24519 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:27.560453 28386 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:27.563043 28536 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.567081 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a35079a106c4cca94cf65ffa93aefbf with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.567504 24522 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.569738 24244 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.571432 28528 ts_tablet_manager.cc:1888] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:27.571571 28528 log.cc:1196] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:27.572397 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:27.586954 28528 tablet_replica.cc:330] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:27.587232 28528 raft_consensus.cc:2229] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.587512 28528 raft_consensus.cc:2258] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.588420 28528 ts_tablet_manager.cc:1875] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:27.593395 28534 ts_tablet_manager.cc:902] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:27.593554 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:27.593976 28428 tablet_replica.cc:330] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:27.594241 28428 raft_consensus.cc:2229] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.594722 28428 raft_consensus.cc:2258] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.596849 28428 ts_tablet_manager.cc:1875] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:27.597632 26611 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:27.598608 28000 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:27.600790 26611 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:27.611343 28544 tablet_replica.cc:330] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:27.611641 28544 raft_consensus.cc:2229] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.612063 28544 raft_consensus.cc:2258] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.616294 28544 ts_tablet_manager.cc:1875] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:27.617391 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:27.619771 28549 ts_tablet_manager.cc:902] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:27.622918 28428 ts_tablet_manager.cc:1888] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:27:27.623076 28428 log.cc:1196] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:27.627920 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 7a35079a106c4cca94cf65ffa93aefbf (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:27.617528 24779 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:27.632988 28150 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:27.634480 28544 ts_tablet_manager.cc:1888] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:27.634627 28544 log.cc:1196] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/a8a9b14a7ff04ddab0d1e1a8e9fcb9b0
I20240502 21:27:27.635568 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 (table rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]) successfully deleted
I20240502 21:27:27.657054 28323 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.657687 28329 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.666105 26743 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.668210 24780 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.675473 24244 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.684271 24651 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:27.687875 24780 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:27.688166 28528 ts_tablet_manager.cc:1888] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:27.688445 28528 log.cc:1196] T 9963dcea478749e0a6e076fac6537fc4 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:27.688584 28323 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:27.690480 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 9963dcea478749e0a6e076fac6537fc4 (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:27.692181 28528 tablet_replica.cc:330] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:27.692417 28528 raft_consensus.cc:2229] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:27.692728 28528 raft_consensus.cc:2258] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:27.699371 28528 ts_tablet_manager.cc:1875] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:27.703943 28530 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:27.707419 28323 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.711818 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:27.757925 24653 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:27.767182 26742 raft_consensus.cc:1203] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 21:27:27.803817 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049bec57d2474c328d2e26e16a4d33a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:27.721537 24388 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.722172 24385 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:27.827715 24651 consensus_queue.cc:237] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:27.816691 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:27.722680 24386 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:27.876523 28438 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20240502 21:27:27.767952 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477d9df7d3a74b8584f3669be3256e00. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:27.971922 28530 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:27:27.971066 24322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 203113us
I20240502 21:27:27.718807 26479 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:27.813213 28549 tablet_copy_client.cc:308] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:27.737445 28534 tablet_copy_client.cc:308] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:27.719586 24779 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.792847 28523 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:27.742900 28528 ts_tablet_manager.cc:1888] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:27.974887 28528 log.cc:1196] T 82fb425a5430426f8030ba5497736b11 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/82fb425a5430426f8030ba5497736b11
I20240502 21:27:27.976150 28523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:34476 (request call id 1) took 1009 ms. Trace:
I20240502 21:27:27.976362 24542 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 53c1992a21224bac883da474f143c1d7 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:27.976277 28523 rpcz_store.cc:276] 0502 21:27:26.966992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:27:26.967048 (+ 56us) service_pool.cc:225] Handling call
0502 21:27:27.427324 (+460276us) log.cc:842] Serialized 499 byte log entry
0502 21:27:27.976101 (+548777us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1182,"raft.run_cpu_time_us":893,"raft.run_wall_time_us":3286,"thread_start_us":185377,"threads_started":1,"wal-append.queue_time_us":162794}
I20240502 21:27:27.976557 24542 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 53c1992a21224bac883da474f143c1d7 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-53c1992a21224bac883da474f143c1d7
W20240502 21:27:27.970260 24587 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.7:36311, user_credentials={real_user=slave}} blocked reactor thread for 166447us
I20240502 21:27:27.977739 28150 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:27.795207 24782 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:27.979024 24519 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:27.979426 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:27.979558 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 6c4aa176fd994cf0b55f304cf9d06fe0 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-6c4aa176fd994cf0b55f304cf9d06fe0
I20240502 21:27:27.980661 28329 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:27.980777 28530 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:27.982379 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cf38bbaaaa340f89d766a843c6bcaf9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:27.783310 24384 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:27.983951 24542 tablet_copy_source_session.cc:205] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:27.715714 24244 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.971246 28438 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:27.855825 24234 catalog_manager.cc:5031] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 21:27:27.984930 28438 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.432s user 0.106s sys 0.094s
I20240502 21:27:27.985441 28534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1992a21224bac883da474f143c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.985769 28438 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:27.977481 26742 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:27.986322 28438 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: NON_PARTICIPANT
I20240502 21:27:27.986598 28438 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:27.976058 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 82fb425a5430426f8030ba5497736b11 (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
I20240502 21:27:27.989687 24651 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:27.990998 24241 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:27.991576 28438 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:27.992956 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-7a35079a106c4cca94cf65ffa93aefbf received from {username='slave'} at 127.16.122.8:33001
I20240502 21:27:27.993088 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-7a35079a106c4cca94cf65ffa93aefbf on tablet 7a35079a106c4cca94cf65ffa93aefbf with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:27.994202 28534 tablet_copy_client.cc:783] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:27.994684 28534 tablet_copy_client.cc:647] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:27.995745 28308 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:27:28.004274 24386 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.005165 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:28.005544 28541 tablet_replica.cc:330] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:28.006038 28541 raft_consensus.cc:2229] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20240502 21:27:28.006453 28541 pending_rounds.cc:62] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Trying to abort 1 pending ops.
I20240502 21:27:28.006611 28541 pending_rounds.cc:68] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023350773787553792 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7a35079a106c4cca94cf65ffa93aefbf" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } } }
I20240502 21:27:28.007135 28541 raft_consensus.cc:2894] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 NON_PARTICIPANT]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 21:27:28.007302 28541 raft_consensus.cc:2866] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:28.007467 28541 raft_consensus.cc:2258] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20240502 21:27:28.009104 28541 ts_tablet_manager.cc:1875] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:28.016887 28541 ts_tablet_manager.cc:1888] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:28.017060 28541 log.cc:1196] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:28.021945 24541 tablet_copy_source_session.cc:205] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.021967 24385 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.024174 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 566929e603d54be4bf931413f0fa1d7d with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:28.026556 28549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4aa176fd994cf0b55f304cf9d06fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.027434 28530 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.028154 24242 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.035629 24519 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.038093 28137 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:28.040495 27637 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.043280 28596 tablet_replica.cc:330] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:28.043597 28596 raft_consensus.cc:2229] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.044030 28596 raft_consensus.cc:2258] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.044160 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.044915 28596 ts_tablet_manager.cc:1875] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:27:28.049019 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477d9df7d3a74b8584f3669be3256e00. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.049875 24782 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.050737 28523 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:28.051360 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477d9df7d3a74b8584f3669be3256e00. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.051445 24653 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.054606 28524 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.054387 28526 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.056519 24653 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.059315 24243 catalog_manager.cc:5484] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.060024 24782 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.066996 28534 tablet_copy_client.cc:523] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.075543 24241 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.078646 24386 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.082314 28596 ts_tablet_manager.cc:1888] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:27:28.082515 28596 log.cc:1196] T 566929e603d54be4bf931413f0fa1d7d P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:28.083681 28549 tablet_copy_client.cc:783] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.084090 28549 tablet_copy_client.cc:647] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.086397 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 566929e603d54be4bf931413f0fa1d7d (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
I20240502 21:27:28.087872 28549 tablet_copy_client.cc:523] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.090265 24651 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:28.096184 28549 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
W20240502 21:27:28.096772 24586 consensus_peers.cc:485] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566929e603d54be4bf931413f0fa1d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.098227 26742 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:28.099107 28530 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:28.099758 28534 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:28.101788 27637 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.102988 24232 catalog_manager.cc:4830] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): tablet 7a35079a106c4cca94cf65ffa93aefbf (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
W20240502 21:27:28.106612 24322 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98b5668e0d504db5a7874f041c14dbfa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.106889 26610 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:28.107640 28524 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:28.109875 24234 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:28.110908 24653 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:28.111712 26742 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.115373 28524 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:28.120169 24782 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:28.120937 28323 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:28.123214 24782 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.132655 24244 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.140513 28523 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.141465 24648 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.157387 24389 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.157265 26610 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.157775 24651 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.169169 28137 tablet_copy_client.cc:308] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:28.170660 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:28.170862 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:28.178310 24410 tablet_copy_source_session.cc:205] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 21:27:28.189644 24586 consensus_peers.cc:485] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566929e603d54be4bf931413f0fa1d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.189872 24241 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
W20240502 21:27:28.191502 24587 consensus_peers.cc:485] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566929e603d54be4bf931413f0fa1d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.193159 24782 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:28.193809 28530 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:28.194041 26742 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:28.197290 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.196770 28530 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.201647 24782 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.206809 28329 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:28.208806 26741 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.210861 24234 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:28.217500 24243 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 9 to 10, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.237103 24389 consensus_queue.cc:237] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.244834 24782 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.245760 24519 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:28.246153 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dd91818e74e42acb276a3864753094d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.246691 28526 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.249507 28526 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.261811 24522 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:28.263341 28137 tablet_copy_client.cc:783] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.263715 28137 tablet_copy_client.cc:647] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.276404 28137 tablet_copy_client.cc:523] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.282892 28137 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:28.292110 28526 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.293203 24782 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.294201 24519 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.303143 24244 catalog_manager.cc:5484] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
W20240502 21:27:28.318363 24455 consensus_peers.cc:485] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0ba1af1d3624d6387bf892873f3f67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.319063 24782 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.319861 28597 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.328984 24651 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.329600 28597 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.334429 28308 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.335429 24779 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.336503 28549 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:28.337275 28549 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:28.337857 28549 ts_tablet_manager.cc:1367] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.242s user 0.066s sys 0.005s
I20240502 21:27:28.343744 24244 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:28.344727 28549 raft_consensus.cc:348] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:28.345043 28549 raft_consensus.cc:729] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:28.345351 28549 consensus_queue.cc:260] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:28.354907 28549 ts_tablet_manager.cc:1398] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240502 21:27:28.356112 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-6c4aa176fd994cf0b55f304cf9d06fe0 received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:28.356261 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-6c4aa176fd994cf0b55f304cf9d06fe0 on tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:28.366485 24651 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.407879 24519 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.410944 24651 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:27:28.411208 24455 consensus_peers.cc:485] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0ba1af1d3624d6387bf892873f3f67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.411659 28527 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.412149 24782 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:27:28.412807 24456 consensus_peers.cc:485] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0ba1af1d3624d6387bf892873f3f67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.413259 28527 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.414340 28527 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.416844 24782 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.418751 24651 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.422518 28658 ts_tablet_manager.cc:902] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:28.423992 24244 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 4 to 5, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.440587 28534 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:28.459139 28534 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:28.459520 28534 ts_tablet_manager.cc:1367] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.360s user 0.046s sys 0.026s
I20240502 21:27:28.460480 28534 raft_consensus.cc:348] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:28.460844 28534 raft_consensus.cc:729] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:28.471179 28534 consensus_queue.cc:260] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:28.485385 26479 raft_consensus.cc:1203] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:28.488713 28534 ts_tablet_manager.cc:1398] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.029s user 0.003s sys 0.001s
I20240502 21:27:28.489565 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-53c1992a21224bac883da474f143c1d7 received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:28.489739 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-53c1992a21224bac883da474f143c1d7 on tablet 53c1992a21224bac883da474f143c1d7 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:28.491161 24519 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:28.496702 26342 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.503293 24782 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.505297 24389 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.507704 28549 ts_tablet_manager.cc:902] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
W20240502 21:27:28.511132 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53c1992a21224bac883da474f143c1d7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.511392 28527 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.511945 28527 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.513437 28277 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.515841 28658 tablet_copy_client.cc:308] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:28.521080 28137 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:28.521164 24384 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:28.521759 28137 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:28.522064 28137 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.239s user 0.056s sys 0.006s
I20240502 21:27:28.523113 28137 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.523506 28137 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:28.523811 28137 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.524737 28137 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:28.525615 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:28.525794 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 on tablet 477d9df7d3a74b8584f3669be3256e00 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:28.530289 28277 raft_consensus.cc:1013] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: attempt to promote peer b77f8f52372241d483f457389fda0f93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 21:27:28.530936 24323 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdf23b7a397441b5947ce6babe3e8f62. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.532171 24779 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.532965 28526 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.540280 24651 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.543574 28150 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.546384 28597 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.549070 28150 raft_consensus.cc:1050] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:28.550196 28150 consensus_queue.cc:237] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:28.548915 24389 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.554066 26342 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.558384 24241 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:28.562191 28527 raft_consensus.cc:1050] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:28.563148 28527 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:28.571108 26344 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:28.572149 28137 tablet_replica.cc:330] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:28.572212 28277 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:28.572432 28137 raft_consensus.cc:2229] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 21:27:28.572674 28137 raft_consensus.cc:2258] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 21:27:28.573817 28137 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:27:28.577963 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53c1992a21224bac883da474f143c1d7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.584164 24384 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:28.585062 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 049bec57d2474c328d2e26e16a4d33a9 from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:55613)
I20240502 21:27:28.585306 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 049bec57d2474c328d2e26e16a4d33a9 from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:55613: session id = 9b0952b24bd8400c9940dbd5e28417b4-049bec57d2474c328d2e26e16a4d33a9
I20240502 21:27:28.589344 24673 tablet_copy_source_session.cc:205] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.590512 24519 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.590920 28691 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:28.591367 28658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049bec57d2474c328d2e26e16a4d33a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.592574 26741 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.593658 28526 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.594477 28526 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.602494 28527 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:28.604655 24648 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:28.605473 28150 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.612763 28277 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.621099 28534 ts_tablet_manager.cc:902] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:28.622896 28658 tablet_copy_client.cc:783] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.624107 28658 tablet_copy_client.cc:647] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.623196 24243 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 5 to 6, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:28.626188 26342 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.630896 28658 tablet_copy_client.cc:523] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.637773 28658 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:28.638486 24780 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:28.641412 28526 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.642679 28701 ts_tablet_manager.cc:902] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:28.657655 24653 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.663028 24243 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:28.665910 24456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.1:36473, user_credentials={real_user=slave}} blocked reactor thread for 46430.3us
I20240502 21:27:28.666882 24384 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.667622 28137 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:28.667855 28137 log.cc:1196] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:28.669018 28137 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
W20240502 21:27:28.669212 24455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 47757us
I20240502 21:27:28.673570 28597 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:28.675514 24782 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.676656 24779 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:28.689929 28150 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.692551 28549 tablet_copy_client.cc:308] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:28.693089 24653 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.697414 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:46751)
I20240502 21:27:28.697682 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 4cf38bbaaaa340f89d766a843c6bcaf9 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:46751: session id = 836775e329f148e5908a1afc54f06428-4cf38bbaaaa340f89d766a843c6bcaf9
I20240502 21:27:28.701901 24673 tablet_copy_source_session.cc:205] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.704111 28549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf38bbaaaa340f89d766a843c6bcaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.706923 24782 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:28.707809 28534 tablet_copy_client.cc:308] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:28.709302 24243 catalog_manager.cc:5484] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:28.711144 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 566929e603d54be4bf931413f0fa1d7d from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:36629)
I20240502 21:27:28.711297 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 566929e603d54be4bf931413f0fa1d7d from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:36629: session id = b77f8f52372241d483f457389fda0f93-566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:28.715838 26741 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.717072 24519 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.722391 24384 consensus_queue.cc:237] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:28.723549 24673 tablet_copy_source_session.cc:205] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.732934 28726 ts_tablet_manager.cc:902] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:28.739145 28534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.739517 28701 tablet_copy_client.cc:308] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:28.740505 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 98b5668e0d504db5a7874f041c14dbfa from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:28.740834 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 98b5668e0d504db5a7874f041c14dbfa from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-98b5668e0d504db5a7874f041c14dbfa
I20240502 21:27:28.725595 24648 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:28.743060 28150 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:28.743978 24410 tablet_copy_source_session.cc:205] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.749959 28524 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.751116 24653 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.761034 28549 tablet_copy_client.cc:783] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.760792 24243 catalog_manager.cc:5484] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.762739 28549 tablet_copy_client.cc:647] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.763391 28701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.766535 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 616e911db4384c17905713adc595ea0a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:28.767079 26742 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:28.768997 28150 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:28.772015 26741 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:28.773722 28549 tablet_copy_client.cc:523] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.778864 28549 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:28.782548 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 616e911db4384c17905713adc595ea0a with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:28.793974 28534 tablet_copy_client.cc:783] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.794384 28534 tablet_copy_client.cc:647] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.800772 28137 tablet_copy_client.cc:308] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:28.801597 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:28.801755 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:28.804701 24410 tablet_copy_source_session.cc:205] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.807902 28738 tablet_replica.cc:330] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:28.808440 28691 tablet_copy_client.cc:308] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:28.808859 28738 raft_consensus.cc:2229] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:28.809237 28738 raft_consensus.cc:2258] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:28.809536 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 477d9df7d3a74b8584f3669be3256e00 from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:28.809712 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 477d9df7d3a74b8584f3669be3256e00 from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:28.814661 24410 tablet_copy_source_session.cc:205] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.818166 28701 tablet_copy_client.cc:783] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.818547 28137 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:28.819655 28742 ts_tablet_manager.cc:902] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:28.820050 24384 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:28.822492 28534 tablet_copy_client.cc:523] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.822876 28741 ts_tablet_manager.cc:902] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:28.827683 28738 ts_tablet_manager.cc:1875] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:27:28.832892 24322 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b6a4a74e0544e0e99e734a145c91d3b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.822149 28691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.834074 28534 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:28.834062 24782 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:28.834100 26610 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:28.834825 28150 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:28.835150 28524 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:28.847873 28523 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.853884 28137 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:27:28.856045 24242 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:28.857550 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.858876 24779 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.868078 28738 ts_tablet_manager.cc:1888] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:28.868906 28738 log.cc:1196] T 616e911db4384c17905713adc595ea0a P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/616e911db4384c17905713adc595ea0a
I20240502 21:27:28.871925 28752 ts_tablet_manager.cc:902] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:28.871742 26609 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:28.872843 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 616e911db4384c17905713adc595ea0a (table rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]) successfully deleted
I20240502 21:27:28.874168 24519 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:28.877478 28137 tablet_copy_client.cc:783] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.878257 28137 tablet_copy_client.cc:647] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.881314 28137 tablet_copy_client.cc:523] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.890738 28726 tablet_copy_client.cc:308] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:28.890961 28701 tablet_copy_client.cc:647] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.891908 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 5dd91818e74e42acb276a3864753094d from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:28.892103 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 5dd91818e74e42acb276a3864753094d from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-5dd91818e74e42acb276a3864753094d
I20240502 21:27:28.896471 24410 tablet_copy_source_session.cc:205] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.898753 28726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd91818e74e42acb276a3864753094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.898790 28691 tablet_copy_client.cc:783] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.902675 28691 tablet_copy_client.cc:647] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.905377 28691 tablet_copy_client.cc:523] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.907271 28137 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:28.912034 24384 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.912842 28597 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.914749 28701 tablet_copy_client.cc:523] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.918008 28741 tablet_copy_client.cc:308] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:28.918953 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet d0ba1af1d3624d6387bf892873f3f67e from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:28.919139 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet d0ba1af1d3624d6387bf892873f3f67e from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-d0ba1af1d3624d6387bf892873f3f67e
I20240502 21:27:28.922128 24541 tablet_copy_source_session.cc:205] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:28.924930 24648 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:28.926015 28277 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:28.928870 28741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ba1af1d3624d6387bf892873f3f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.931005 28701 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:28.936604 28698 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.937336 24384 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.950976 24653 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.960335 24242 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:28.969431 28741 tablet_copy_client.cc:783] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.970813 28741 tablet_copy_client.cc:647] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.973345 28741 tablet_copy_client.cc:523] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 21:27:28.976217 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f912eb05c574368909e0b3a8edf058d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:28.983392 28741 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:28.992650 28726 tablet_copy_client.cc:783] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:28.993391 28726 tablet_copy_client.cc:647] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:28.997313 28726 tablet_copy_client.cc:523] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:28.999184 28742 tablet_copy_client.cc:308] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:29.000211 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 566929e603d54be4bf931413f0fa1d7d from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:55613)
I20240502 21:27:29.000437 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 566929e603d54be4bf931413f0fa1d7d from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:55613: session id = 9b0952b24bd8400c9940dbd5e28417b4-566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:29.001019 28691 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:29.003615 24673 tablet_copy_source_session.cc:205] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:29.011566 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.014302 28726 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:29.019518 28752 tablet_copy_client.cc:308] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:29.022895 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet d0ba1af1d3624d6387bf892873f3f67e from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:35031)
I20240502 21:27:29.023082 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet d0ba1af1d3624d6387bf892873f3f67e from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:35031: session id = 9b0952b24bd8400c9940dbd5e28417b4-d0ba1af1d3624d6387bf892873f3f67e
I20240502 21:27:29.032621 24541 tablet_copy_source_session.cc:205] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:29.034680 28752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ba1af1d3624d6387bf892873f3f67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.040257 24519 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:29.052764 28752 tablet_copy_client.cc:783] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
W20240502 21:27:29.058951 24455 consensus_peers.cc:485] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc717a1b12e4380a4107e6deae2b837. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.060196 26741 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:29.060845 24653 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:29.061602 28698 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:29.063055 28698 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:29.065282 28752 tablet_copy_client.cc:647] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.059314 28742 tablet_copy_client.cc:783] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:29.068992 28742 tablet_copy_client.cc:647] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.069769 28698 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.071962 28752 tablet_copy_client.cc:523] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.075698 24653 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.086877 24243 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.090765 28597 raft_consensus.cc:1050] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:29.091729 28597 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:29.099431 28809 ts_tablet_manager.cc:902] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:29.101300 28742 tablet_copy_client.cc:523] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.102402 28752 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:29.111904 28742 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:29.117319 26741 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.121588 26479 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.122294 28277 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:29.124496 24389 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.126601 28277 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:29.130699 28658 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.131210 28809 tablet_copy_client.cc:308] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:29.131889 28658 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:29.132032 24410 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet bdf23b7a397441b5947ce6babe3e8f62 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:45315)
I20240502 21:27:29.132277 24410 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet bdf23b7a397441b5947ce6babe3e8f62 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:45315: session id = 836775e329f148e5908a1afc54f06428-bdf23b7a397441b5947ce6babe3e8f62
I20240502 21:27:29.132349 28658 ts_tablet_manager.cc:1367] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.495s user 0.052s sys 0.020s
I20240502 21:27:29.133678 28658 raft_consensus.cc:348] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.134136 28658 raft_consensus.cc:729] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:29.134438 28658 consensus_queue.cc:260] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.136847 24782 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.138396 28277 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:29.142575 24410 tablet_copy_source_session.cc:205] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:29.149561 28809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.161458 28824 ts_tablet_manager.cc:902] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:29.163532 28698 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.165478 28658 ts_tablet_manager.cc:1398] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240502 21:27:29.169140 24389 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.170830 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-049bec57d2474c328d2e26e16a4d33a9 received from {username='slave'} at 127.16.122.7:55613
I20240502 21:27:29.170969 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-049bec57d2474c328d2e26e16a4d33a9 on tablet 049bec57d2474c328d2e26e16a4d33a9 with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:29.176455 26476 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.178942 24779 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:29.186019 24241 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.193670 24519 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:29.205879 24384 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:29.206964 28832 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:29.209460 28597 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.210141 24389 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.213793 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:29.217681 24241 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.223666 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:29.224447 26479 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:29.225558 28832 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:29.227483 26479 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:29.235414 24389 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:29.240463 28824 tablet_copy_client.cc:308] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:29.241377 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 53c1992a21224bac883da474f143c1d7 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:34905)
I20240502 21:27:29.241602 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 53c1992a21224bac883da474f143c1d7 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:34905: session id = 1769ee000eb24298a367a0de1209e6e0-53c1992a21224bac883da474f143c1d7
W20240502 21:27:29.242620 24322 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdf23b7a397441b5947ce6babe3e8f62. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.243669 24648 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.244587 24780 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.244815 24541 tablet_copy_source_session.cc:205] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:29.246917 28824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1992a21224bac883da474f143c1d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.246222 28150 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 21:27:29.247651 28150 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:29.279510 28824 tablet_copy_client.cc:783] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:29.280181 28824 tablet_copy_client.cc:647] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.279867 24780 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.280422 28847 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.283207 28824 tablet_copy_client.cc:523] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.294018 24247 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 4 to 5, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.304354 28824 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
W20240502 21:27:29.336364 24323 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.336691 28809 tablet_copy_client.cc:783] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:29.338044 28809 tablet_copy_client.cc:647] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.344209 28549 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.344794 28549 tablet_bootstrap.cc:492] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:29.345108 28549 ts_tablet_manager.cc:1367] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.566s user 0.063s sys 0.002s
I20240502 21:27:29.345990 28809 tablet_copy_client.cc:523] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.349146 28852 ts_tablet_manager.cc:902] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:29.350138 28809 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:29.356348 28549 raft_consensus.cc:348] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:29.366508 28549 raft_consensus.cc:729] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:29.367005 28549 consensus_queue.cc:260] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:29.368906 28549 ts_tablet_manager.cc:1398] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.024s user 0.002s sys 0.003s
I20240502 21:27:29.369989 28658 ts_tablet_manager.cc:902] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:29.370033 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-4cf38bbaaaa340f89d766a843c6bcaf9 received from {username='slave'} at 127.16.122.6:46751
I20240502 21:27:29.370268 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 836775e329f148e5908a1afc54f06428-4cf38bbaaaa340f89d766a843c6bcaf9 on tablet 4cf38bbaaaa340f89d766a843c6bcaf9 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:29.383852 28534 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 21:27:29.389616 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e847a796029d4edea79694f5402a291a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.391178 24653 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:29.392155 28536 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:29.406903 28534 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:29.407375 28534 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.573s user 0.059s sys 0.012s
I20240502 21:27:29.409474 28534 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.390529 26342 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:29.410809 28534 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:29.411221 28534 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.412600 24519 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:29.414706 28137 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 21:27:29.415910 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:29.417201 26342 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:29.418098 28597 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:29.419687 28536 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:29.423336 28597 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.430383 26342 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.439137 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53c1992a21224bac883da474f143c1d7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:29.441895 28137 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:29.442278 28137 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.540s user 0.053s sys 0.012s
I20240502 21:27:29.448037 24651 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.450167 24779 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:29.447242 28854 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.444593 28137 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.453454 28137 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:29.453727 28137 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.455344 28597 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:29.458976 24243 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.465348 28137 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20240502 21:27:29.483835 28701 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.492484 28701 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:29.493877 28701 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.567s user 0.066s sys 0.001s
I20240502 21:27:29.497056 26609 raft_consensus.cc:1203] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:29.498077 28701 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:29.499801 28701 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:29.502393 28701 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:29.505177 28701 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 21:27:29.495396 24782 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.524096 26476 raft_consensus.cc:1203] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:29.532685 28726 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.540580 28726 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:29.533509 28691 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.543078 28691 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:29.543490 28691 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.549s user 0.042s sys 0.020s
I20240502 21:27:29.544450 28691 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.545464 28741 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.556836 28809 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.557495 28809 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:29.557904 28809 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.208s user 0.056s sys 0.009s
I20240502 21:27:29.559216 28809 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:29.560257 28809 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:29.560607 28841 tablet_replica.cc:330] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:29.560707 28809 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:29.560981 28841 raft_consensus.cc:2229] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.561280 28841 pending_rounds.cc:62] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Trying to abort 1 pending ops.
I20240502 21:27:29.561403 28841 pending_rounds.cc:68] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350780277334016 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6c4aa176fd994cf0b55f304cf9d06fe0" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } } }
I20240502 21:27:29.562003 28841 raft_consensus.cc:2894] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:27:29.562206 28841 raft_consensus.cc:2866] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:29.562698 28841 raft_consensus.cc:2258] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.562753 28809 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:29.561097 24243 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:29.564585 28741 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:29.565438 28691 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:29.565711 28691 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.566413 28691 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240502 21:27:29.545866 28726 ts_tablet_manager.cc:1367] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.532s user 0.051s sys 0.009s
I20240502 21:27:29.567581 28726 raft_consensus.cc:348] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.567961 28726 raft_consensus.cc:729] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:29.568192 28726 consensus_queue.cc:260] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.568790 28726 ts_tablet_manager.cc:1398] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:29.571787 28658 tablet_copy_client.cc:308] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:29.572619 28741 ts_tablet_manager.cc:1367] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.582s user 0.049s sys 0.013s
I20240502 21:27:29.572590 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 8f912eb05c574368909e0b3a8edf058d from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:35031)
I20240502 21:27:29.572813 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 8f912eb05c574368909e0b3a8edf058d from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:35031: session id = 9b0952b24bd8400c9940dbd5e28417b4-8f912eb05c574368909e0b3a8edf058d
I20240502 21:27:29.574883 28741 raft_consensus.cc:348] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.575627 24410 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-bdf23b7a397441b5947ce6babe3e8f62 received from {username='slave'} at 127.16.122.6:45315
I20240502 21:27:29.575799 24410 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 836775e329f148e5908a1afc54f06428-bdf23b7a397441b5947ce6babe3e8f62 on tablet bdf23b7a397441b5947ce6babe3e8f62 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:29.576262 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c1992a21224bac883da474f143c1d7 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:29.576500 24409 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-477d9df7d3a74b8584f3669be3256e00 received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:29.576648 24409 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-477d9df7d3a74b8584f3669be3256e00 on tablet 477d9df7d3a74b8584f3669be3256e00 with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:29.577179 24541 tablet_copy_source_session.cc:205] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 21:27:29.577761 24324 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.3:38427, user_credentials={real_user=slave}} blocked reactor thread for 313245us
I20240502 21:27:29.578706 24651 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.592984 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-5dd91818e74e42acb276a3864753094d received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:29.593240 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-5dd91818e74e42acb276a3864753094d on tablet 5dd91818e74e42acb276a3864753094d with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:29.594401 28658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f912eb05c574368909e0b3a8edf058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.599968 28887 tablet_replica.cc:330] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:29.601421 28887 raft_consensus.cc:2229] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:29.601804 28887 raft_consensus.cc:2258] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:29.607276 28887 ts_tablet_manager.cc:1875] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:29.609846 28741 raft_consensus.cc:729] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:29.610175 28741 consensus_queue.cc:260] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.612847 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:29.613044 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 on tablet 477d9df7d3a74b8584f3669be3256e00 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:29.616585 28658 tablet_copy_client.cc:783] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:29.616839 28658 tablet_copy_client.cc:647] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.619282 28658 tablet_copy_client.cc:523] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.619355 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-98b5668e0d504db5a7874f041c14dbfa received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:29.619520 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-98b5668e0d504db5a7874f041c14dbfa on tablet 98b5668e0d504db5a7874f041c14dbfa with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:29.627457 28824 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.628681 28824 tablet_bootstrap.cc:492] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:29.628831 28887 ts_tablet_manager.cc:1888] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:27:29.629061 28887 log.cc:1196] T 53c1992a21224bac883da474f143c1d7 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/53c1992a21224bac883da474f143c1d7
I20240502 21:27:29.629398 28824 ts_tablet_manager.cc:1367] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.325s user 0.058s sys 0.020s
I20240502 21:27:29.630949 28824 raft_consensus.cc:348] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:29.631608 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 53c1992a21224bac883da474f143c1d7 (table rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]) successfully deleted
I20240502 21:27:29.632799 28824 raft_consensus.cc:729] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:29.633407 28824 consensus_queue.cc:260] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:29.634217 28742 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.634812 28742 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:29.635298 28742 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.523s user 0.062s sys 0.004s
I20240502 21:27:29.636263 28742 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.636881 28742 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:29.638098 28658 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:29.638247 28742 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.641788 28742 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:29.642599 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-566929e603d54be4bf931413f0fa1d7d received from {username='slave'} at 127.16.122.7:55613
I20240502 21:27:29.642740 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-566929e603d54be4bf931413f0fa1d7d on tablet 566929e603d54be4bf931413f0fa1d7d with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:29.644291 26741 raft_consensus.cc:1203] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:29.648583 28824 ts_tablet_manager.cc:1398] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 21:27:29.653668 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-53c1992a21224bac883da474f143c1d7 received from {username='slave'} at 127.16.122.8:34905
I20240502 21:27:29.653831 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-53c1992a21224bac883da474f143c1d7 on tablet 53c1992a21224bac883da474f143c1d7 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:29.655565 28741 ts_tablet_manager.cc:1398] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.083s user 0.003s sys 0.000s
I20240502 21:27:29.655864 28879 ts_tablet_manager.cc:902] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:29.656019 28891 ts_tablet_manager.cc:902] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:29.660254 28534 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.252s user 0.027s sys 0.041s
I20240502 21:27:29.662111 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-566929e603d54be4bf931413f0fa1d7d received from {username='slave'} at 127.16.122.5:36629
I20240502 21:27:29.662252 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session b77f8f52372241d483f457389fda0f93-566929e603d54be4bf931413f0fa1d7d on tablet 566929e603d54be4bf931413f0fa1d7d with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:29.663899 28752 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.664399 28752 tablet_bootstrap.cc:492] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:29.664758 28752 ts_tablet_manager.cc:1367] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.563s user 0.053s sys 0.010s
I20240502 21:27:29.665750 28752 raft_consensus.cc:348] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.666129 28752 raft_consensus.cc:729] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:29.670498 28752 consensus_queue.cc:260] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.671392 28752 ts_tablet_manager.cc:1398] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:29.676772 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-d0ba1af1d3624d6387bf892873f3f67e received from {username='slave'} at 127.16.122.7:35031
I20240502 21:27:29.676978 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-d0ba1af1d3624d6387bf892873f3f67e on tablet d0ba1af1d3624d6387bf892873f3f67e with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:29.696663 28841 ts_tablet_manager.cc:1875] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:29.707810 26345 raft_consensus.cc:1203] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:29.711503 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-d0ba1af1d3624d6387bf892873f3f67e received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:29.711699 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-d0ba1af1d3624d6387bf892873f3f67e on tablet d0ba1af1d3624d6387bf892873f3f67e with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:29.690153 26342 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.754513 28658 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:29.755075 28658 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:29.755409 28658 ts_tablet_manager.cc:1367] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.117s user 0.067s sys 0.000s
I20240502 21:27:29.756278 28658 raft_consensus.cc:348] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.756623 28658 raft_consensus.cc:729] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:29.756914 28658 consensus_queue.cc:260] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:29.759028 28658 ts_tablet_manager.cc:1398] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:29.759917 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-8f912eb05c574368909e0b3a8edf058d received from {username='slave'} at 127.16.122.7:35031
I20240502 21:27:29.760097 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-8f912eb05c574368909e0b3a8edf058d on tablet 8f912eb05c574368909e0b3a8edf058d with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:29.764982 26609 raft_consensus.cc:1203] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:29.790530 28852 tablet_copy_client.cc:308] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:29.828462 26741 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:29.849169 26344 raft_consensus.cc:1203] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:27:29.849844 28879 tablet_copy_client.cc:308] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:29.850768 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 0fc717a1b12e4380a4107e6deae2b837 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:29.851032 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 0fc717a1b12e4380a4107e6deae2b837 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:29.854131 24541 tablet_copy_source_session.cc:205] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:29.855582 28879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.857968 26609 raft_consensus.cc:1203] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:29.859669 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 2b6a4a74e0544e0e99e734a145c91d3b from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:41749)
I20240502 21:27:29.859862 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 2b6a4a74e0544e0e99e734a145c91d3b from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:41749: session id = b77f8f52372241d483f457389fda0f93-2b6a4a74e0544e0e99e734a145c91d3b
I20240502 21:27:29.863197 28879 tablet_copy_client.cc:783] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:29.863539 28879 tablet_copy_client.cc:647] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.864411 24408 tablet_copy_source_session.cc:205] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:29.865782 28852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6a4a74e0544e0e99e734a145c91d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.866791 28879 tablet_copy_client.cc:523] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.872944 26609 raft_consensus.cc:1203] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:29.878154 28841 ts_tablet_manager.cc:1888] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:29.879019 28841 log.cc:1196] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c4aa176fd994cf0b55f304cf9d06fe0
I20240502 21:27:29.881002 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 6c4aa176fd994cf0b55f304cf9d06fe0 (table rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]) successfully deleted
I20240502 21:27:29.886329 26476 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:29.889261 28879 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:29.892308 28852 tablet_copy_client.cc:783] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:29.892604 28852 tablet_copy_client.cc:647] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:29.898953 28852 tablet_copy_client.cc:523] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:29.904129 28852 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:29.913563 26609 raft_consensus.cc:1203] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:27:29.919340 24782 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:29.964331 26344 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:29.966387 28856 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:29.971359 24519 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:29.971788 28329 raft_consensus.cc:1050] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:29.972405 28856 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:29.972910 28329 consensus_queue.cc:237] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:29.975842 26609 raft_consensus.cc:1203] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:27:29.981328 24522 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.987572 26479 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.988562 27637 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:29.992312 27637 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:29.995544 28824 ts_tablet_manager.cc:902] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:29.998430 24389 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:29.999431 27637 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:30.003957 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b2661e0b5fe4a0b94c150b14b8fd9ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:30.024252 26741 raft_consensus.cc:1203] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:27:30.034369 28879 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:30.034947 28879 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:30.035290 28879 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.165s user 0.049s sys 0.016s
I20240502 21:27:30.039381 28879 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:30.039803 28879 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:30.041463 28879 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:30.041425 28536 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:30.042311 24520 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:30.048032 28879 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:30.048777 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-0fc717a1b12e4380a4107e6deae2b837 received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:30.048921 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-0fc717a1b12e4380a4107e6deae2b837 on tablet 0fc717a1b12e4380a4107e6deae2b837 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:30.054322 24244 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:30.070950 28891 tablet_copy_client.cc:308] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:30.071800 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet bdf23b7a397441b5947ce6babe3e8f62 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:41749)
I20240502 21:27:30.072016 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet bdf23b7a397441b5947ce6babe3e8f62 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:41749: session id = b77f8f52372241d483f457389fda0f93-bdf23b7a397441b5947ce6babe3e8f62
I20240502 21:27:30.078639 24408 tablet_copy_source_session.cc:205] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:30.112912 28891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.124230 27637 raft_consensus.cc:1050] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:30.126246 27637 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.132193 28845 raft_consensus.cc:1050] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:30.133862 28845 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:30.147336 28530 raft_consensus.cc:1050] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:30.148530 28530 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:30.150429 28891 tablet_copy_client.cc:783] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:30.156461 24522 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:30.166802 28891 tablet_copy_client.cc:647] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
W20240502 21:27:30.167948 28932 log.cc:924] Time spent T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20240502 21:27:30.171015 28891 tablet_copy_client.cc:523] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:30.156312 28329 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:30.172849 26344 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:30.178112 28824 tablet_copy_client.cc:308] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:30.180306 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet e847a796029d4edea79694f5402a291a from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:33001)
I20240502 21:27:30.180521 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet e847a796029d4edea79694f5402a291a from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:33001: session id = 1769ee000eb24298a367a0de1209e6e0-e847a796029d4edea79694f5402a291a
I20240502 21:27:30.181759 28891 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:30.183861 24804 tablet_copy_source_session.cc:205] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:30.185781 28824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e847a796029d4edea79694f5402a291a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.188846 24389 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:30.196979 26479 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:30.201488 24242 catalog_manager.cc:5484] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:30.213888 28824 tablet_copy_client.cc:783] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:30.214164 28824 tablet_copy_client.cc:647] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:30.216369 28824 tablet_copy_client.cc:523] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:30.218552 24651 consensus_queue.cc:237] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:30.221169 26479 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.222240 24389 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.222301 28843 raft_consensus.cc:1050] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:30.222887 28962 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.223336 28843 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:30.225795 28962 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
W20240502 21:27:30.226963 24322 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:30.228564 26476 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.230742 28524 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.230365 24389 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:30.233588 28949 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.235599 26479 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:30.243110 24651 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.243800 28524 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.245003 28700 raft_consensus.cc:1050] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:30.245935 28700 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:30.246711 28524 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.250891 26741 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.251936 26344 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.252715 28832 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.252647 24653 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.253229 28832 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.255364 24782 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.255972 28597 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.258960 28597 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.259665 24779 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.265812 26609 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.265288 26741 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.267130 24241 catalog_manager.cc:5484] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.267416 28848 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.267994 26476 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.268460 26609 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.270737 28949 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 21:27:30.271718 28949 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.275048 24653 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.275159 24243 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.275962 28848 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.278790 26344 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:30.283069 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:30.284363 26342 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.288754 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:30.292167 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:30.292933 28962 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:30.296597 26742 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.303040 28845 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.308449 24779 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.309285 28848 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:27:30.313144 24587 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.7:36311, user_credentials={real_user=slave}} blocked reactor thread for 44189.1us
I20240502 21:27:30.318194 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:30.323585 28832 raft_consensus.cc:1050] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:30.324946 28832 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.325883 24389 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:30.326838 26479 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.330546 24522 consensus_queue.cc:237] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:30.333771 28994 tablet_replica.cc:330] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:30.334157 26342 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:30.334241 26743 raft_consensus.cc:1261] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:30.334913 28994 raft_consensus.cc:2229] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.335011 28700 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:30.335177 28994 raft_consensus.cc:2258] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.327183 28848 raft_consensus.cc:1050] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:30.336555 28848 consensus_queue.cc:237] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.341697 28700 consensus_queue.cc:1035] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:30.344672 28994 ts_tablet_manager.cc:1875] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:30.347839 28824 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:30.353178 26612 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:30.356472 24779 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.357596 28996 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.359201 28843 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:30.362219 26609 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:30.363401 24782 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:30.364065 26342 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.372164 28845 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.373077 24247 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.374181 24653 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.381703 28597 raft_consensus.cc:1013] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: attempt to promote peer b77f8f52372241d483f457389fda0f93: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:27:30.395749 28329 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:30.399560 28700 raft_consensus.cc:1050] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:30.400354 24651 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.400544 28700 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.402947 28949 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:30.403256 29000 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.403550 28949 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:30.405104 24782 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:30.405879 28848 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:30.415808 24651 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:30.415879 26609 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:30.418694 24389 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:30.452589 28949 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.462939 24244 catalog_manager.cc:5484] T 53c1992a21224bac883da474f143c1d7 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.462853 24247 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.408663 28994 ts_tablet_manager.cc:1888] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:30.467080 28994 log.cc:1196] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/4cf38bbaaaa340f89d766a843c6bcaf9
I20240502 21:27:30.418097 28700 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:30.468590 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 4cf38bbaaaa340f89d766a843c6bcaf9 (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
I20240502 21:27:30.469077 29008 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:30.471043 28975 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:30.473776 27637 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:30.476284 26609 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.481060 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c1992a21224bac883da474f143c1d7 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:30.495581 24247 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.500662 28974 raft_consensus.cc:1050] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:30.507155 24389 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.514330 26344 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:30.502137 28974 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
W20240502 21:27:30.527273 24455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 147572us
I20240502 21:27:30.528548 26342 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.529176 26742 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.536945 24649 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.540459 26611 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.549789 24783 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.553126 26476 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.555239 24242 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.557701 26742 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:30.569031 24522 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:30.570470 26342 raft_consensus.cc:1203] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 21:27:30.570168 26742 raft_consensus.cc:2932] T 53c1992a21224bac883da474f143c1d7 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.574311 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:30.575395 28998 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:30.578979 28975 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.579444 26476 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:30.582012 29041 ts_tablet_manager.cc:902] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:30.584323 29027 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.585318 29027 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.592413 24762 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:30.597183 28962 raft_consensus.cc:1050] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:30.603817 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:30.606415 28962 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.609352 28988 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.611775 27637 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:30.612875 26612 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:30.616860 26344 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:30.617635 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:30.619877 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:30.620762 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53c1992a21224bac883da474f143c1d7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:30.620800 26610 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:30.625749 28979 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:27:30.626863 24455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 48012.7us
I20240502 21:27:30.627143 28979 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:30.633415 26609 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.639530 24781 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:30.644979 28530 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20240502 21:27:30.643891 24779 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:30.646404 24782 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.648294 28949 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.654248 24384 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:30.658207 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:30.658696 26741 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:30.661209 26611 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:30.674496 28998 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:30.675712 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.676796 28988 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.677652 26607 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.681804 26741 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.682495 24242 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.683092 28998 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.686393 28843 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.686767 24244 catalog_manager.cc:5484] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:30.688843 29032 tablet_replica.cc:330] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:30.689179 29032 raft_consensus.cc:2229] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.690498 29032 raft_consensus.cc:2258] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.691749 29027 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.699301 24648 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.699234 24244 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.699822 24389 consensus_queue.cc:237] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.702692 26741 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.706549 26608 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.707836 26479 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.709525 28848 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.710033 28597 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.710984 24784 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.717890 24651 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.720188 26742 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:30.720099 24241 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.720850 28530 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:30.722676 29032 ts_tablet_manager.cc:1875] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:30.723996 28530 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.724997 24244 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.729039 26743 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.731043 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:30.742094 24649 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.743151 28988 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.746245 24783 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.747118 28988 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.753453 29004 raft_consensus.cc:1050] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:30.754552 29004 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.754766 24389 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.761157 28848 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.766142 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:30.766192 26342 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.767963 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5dd91818e74e42acb276a3864753094d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:30.769280 26606 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.773114 26605 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:30.774068 29016 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.775776 26607 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.780659 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98b5668e0d504db5a7874f041c14dbfa with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:30.782219 29041 tablet_copy_client.cc:308] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:30.783115 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34643)
I20240502 21:27:30.783335 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34643: session id = 836775e329f148e5908a1afc54f06428-7b2661e0b5fe4a0b94c150b14b8fd9ec
I20240502 21:27:30.786203 24804 tablet_copy_source_session.cc:205] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:30.786814 24240 catalog_manager.cc:5484] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:30.789959 29041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.796473 26607 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.796531 26606 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.807233 26610 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:30.808019 24777 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:30.810791 28845 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:30.811703 26344 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:30.811856 24781 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.812515 29016 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:30.812271 26610 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:30.816936 29032 ts_tablet_manager.cc:1888] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:30.817088 29032 log.cc:1196] T 53c1992a21224bac883da474f143c1d7 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/53c1992a21224bac883da474f143c1d7
I20240502 21:27:30.818156 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 53c1992a21224bac883da474f143c1d7 (table rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]) successfully deleted
I20240502 21:27:30.821348 29016 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:30.823045 24778 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.825309 28988 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:30.826754 28988 raft_consensus.cc:1013] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: attempt to promote peer 1769ee000eb24298a367a0de1209e6e0: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:27:30.829360 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 049bec57d2474c328d2e26e16a4d33a9 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:30.842095 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd91818e74e42acb276a3864753094d with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:30.842576 28994 tablet_replica.cc:330] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:30.842860 28994 raft_consensus.cc:2229] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.843127 28994 pending_rounds.cc:62] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Trying to abort 1 pending ops.
I20240502 21:27:30.842828 24240 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 11 to 12, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.843189 28994 pending_rounds.cc:68] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350785376088064 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5dd91818e74e42acb276a3864753094d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } } }
I20240502 21:27:30.843698 28994 raft_consensus.cc:2894] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:27:30.843874 28994 raft_consensus.cc:2866] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:30.844101 28994 raft_consensus.cc:2258] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.856442 24389 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:30.862063 26479 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:30.863337 28988 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:30.865972 28988 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.867869 26476 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.871450 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:30.871539 29016 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.876420 24782 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.879794 28994 ts_tablet_manager.cc:1875] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:30.885943 24522 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.888005 28994 ts_tablet_manager.cc:1888] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:30.891593 26612 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.899510 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:30.903386 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdf23b7a397441b5947ce6babe3e8f62 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:30.904296 24778 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.914745 26606 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:30.916783 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:30.921401 28524 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.927291 26606 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.930534 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b5668e0d504db5a7874f041c14dbfa with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:30.937666 24649 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:30.939303 24649 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:30.940223 28845 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:30.942440 24649 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.956010 24649 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:30.903162 24241 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 6 to 7, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:30.960147 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:30.961582 28530 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:30.964929 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf23b7a397441b5947ce6babe3e8f62 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:30.965456 29032 tablet_replica.cc:330] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:30.965716 29032 raft_consensus.cc:2229] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.966055 29032 raft_consensus.cc:2258] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.967358 29032 ts_tablet_manager.cc:1875] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:30.967164 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.972219 24244 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:30.975423 29032 ts_tablet_manager.cc:1888] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:27:30.975656 29032 log.cc:1196] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/bdf23b7a397441b5947ce6babe3e8f62
I20240502 21:27:30.976763 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet bdf23b7a397441b5947ce6babe3e8f62 (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
I20240502 21:27:30.987208 24631 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:30.993110 29004 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20240502 21:27:30.994230 29076 tablet_replica.cc:330] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:30.994532 29076 raft_consensus.cc:2229] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:30.994956 29076 raft_consensus.cc:2258] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:30.994737 28700 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20240502 21:27:30.996191 29076 ts_tablet_manager.cc:1875] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:30.996014 24648 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:30.997339 24651 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.013447 28994 log.cc:1196] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/5dd91818e74e42acb276a3864753094d
I20240502 21:27:31.023758 28700 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.030182 24246 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.034485 26741 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.036728 26342 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.039022 29041 tablet_copy_client.cc:783] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:31.046622 29041 tablet_copy_client.cc:647] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:31.048676 26344 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:31.050429 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:31.055090 29041 tablet_copy_client.cc:523] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:31.058215 24389 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:31.060511 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f912eb05c574368909e0b3a8edf058d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 21:27:31.059720 24322 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:31.064868 29004 raft_consensus.cc:1050] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:31.065670 29004 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:31.071630 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 5dd91818e74e42acb276a3864753094d (table rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]) successfully deleted
I20240502 21:27:31.070909 29041 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:31.073187 26609 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.074391 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.070151 29076 ts_tablet_manager.cc:1888] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:31.077360 29076 log.cc:1196] T 049bec57d2474c328d2e26e16a4d33a9 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/049bec57d2474c328d2e26e16a4d33a9
I20240502 21:27:31.079229 26342 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.080143 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 049bec57d2474c328d2e26e16a4d33a9 (table rebalance_test_table_6 [id=7d3c1d066dcb423d87e5bd15cd684ec7]) successfully deleted
I20240502 21:27:31.085415 28998 raft_consensus.cc:1050] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:31.086570 28998 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:31.088696 29027 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:31.090545 26610 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:31.091277 28843 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:31.101063 24778 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:31.101954 28524 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:31.106492 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:31.107431 28843 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.114118 26610 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.122236 24778 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.131808 26606 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:31.138058 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.143877 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:31.147010 24322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 62411.9us
I20240502 21:27:31.150265 26741 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:31.151132 28700 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:31.153108 26342 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:31.153802 29016 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:31.154748 26742 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:31.162127 28891 log.cc:924] Time spent T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Append to log took a long time: real 0.216s user 0.003s sys 0.000s
W20240502 21:27:31.167098 29051 log.cc:924] Time spent T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20240502 21:27:31.167672 28852 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:31.168486 28852 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:31.168936 28852 ts_tablet_manager.cc:1367] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 1.265s user 0.064s sys 0.004s
I20240502 21:27:31.168886 28979 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
I20240502 21:27:31.170585 28852 raft_consensus.cc:348] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:31.171160 28852 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:31.171502 28852 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:31.172456 28852 ts_tablet_manager.cc:1398] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:31.173345 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-2b6a4a74e0544e0e99e734a145c91d3b received from {username='slave'} at 127.16.122.5:41749
I20240502 21:27:31.173511 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session b77f8f52372241d483f457389fda0f93-2b6a4a74e0544e0e99e734a145c91d3b on tablet 2b6a4a74e0544e0e99e734a145c91d3b with peer b77f8f52372241d483f457389fda0f93
W20240502 21:27:31.181452 24233 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240502 21:27:31.183610 27637 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.013s
I20240502 21:27:31.184096 24244 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 6 to 7, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:31.185446 24651 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:31.185735 27637 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.187824 29027 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:31.190335 24649 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:31.191051 28848 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:31.192924 26742 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.186293 26610 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.195127 26344 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.201553 29083 tablet_replica.cc:330] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:31.205392 29083 raft_consensus.cc:2229] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:31.205546 24389 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.206374 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:31.207034 29083 raft_consensus.cc:2258] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:31.201936 24649 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.222146 28845 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:31.228163 29083 ts_tablet_manager.cc:1875] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:31.231181 26742 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.242410 24522 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:31.245450 26345 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.248405 24242 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 12 to 13, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.249171 28700 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.253156 24778 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.254268 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:31.255172 24241 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.255638 24389 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:31.258209 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 566929e603d54be4bf931413f0fa1d7d with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:31.258805 29032 tablet_replica.cc:330] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:31.259020 29032 raft_consensus.cc:2229] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:31.259128 26606 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.259346 29032 raft_consensus.cc:2258] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:31.260799 29032 ts_tablet_manager.cc:1875] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:31.263211 28848 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.267735 26742 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.270428 28524 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.271283 26609 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.271961 24778 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.272707 28845 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.274911 29027 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.275509 29027 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.276194 28845 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.281387 24782 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.288924 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f912eb05c574368909e0b3a8edf058d with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:31.297526 26609 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.302464 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:31.305683 24241 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:31.312974 28700 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.319106 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0ba1af1d3624d6387bf892873f3f67e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:31.321660 26342 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.322603 24247 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.328562 24389 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:31.329535 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ba1af1d3624d6387bf892873f3f67e with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:31.330145 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 477d9df7d3a74b8584f3669be3256e00 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:31.331171 26609 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:31.332194 28845 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:31.333988 24651 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:31.335063 28845 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.338716 29027 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:31.339301 26606 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.345156 26345 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:31.350334 24240 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 8 to 9, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.353119 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:31.353832 24242 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 9 to 10, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:31.361003 24782 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:31.361868 28845 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:31.364203 24782 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.386124 29041 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:31.386765 29041 tablet_bootstrap.cc:492] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:31.387102 29041 ts_tablet_manager.cc:1367] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.316s user 0.074s sys 0.036s
W20240502 21:27:31.387406 24456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.4:33687, user_credentials={real_user=slave}} blocked reactor thread for 75382.4us
I20240502 21:27:31.388078 29041 raft_consensus.cc:348] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:31.388392 29041 raft_consensus.cc:729] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:31.389050 24782 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.391479 26344 raft_consensus.cc:1203] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:31.404321 29041 consensus_queue.cc:260] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:31.406507 26612 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.428872 26612 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:31.435237 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 477d9df7d3a74b8584f3669be3256e00 with delete_type TABLET_DATA_TOMBSTONED (TS 1769ee000eb24298a367a0de1209e6e0 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:31.451610 26741 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:31.461072 29107 tablet_replica.cc:330] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:31.461539 29107 raft_consensus.cc:2229] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:31.461923 29107 pending_rounds.cc:62] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Trying to abort 1 pending ops.
I20240502 21:27:31.462046 29107 pending_rounds.cc:68] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023350789140205568 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "477d9df7d3a74b8584f3669be3256e00" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } } }
I20240502 21:27:31.462841 29107 raft_consensus.cc:2894] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 21:27:31.463043 29107 raft_consensus.cc:2866] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:31.463218 29107 raft_consensus.cc:2258] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:31.491000 29041 ts_tablet_manager.cc:1398] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.104s user 0.020s sys 0.017s
I20240502 21:27:31.492282 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-7b2661e0b5fe4a0b94c150b14b8fd9ec received from {username='slave'} at 127.16.122.6:34643
I20240502 21:27:31.492439 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 836775e329f148e5908a1afc54f06428-7b2661e0b5fe4a0b94c150b14b8fd9ec on tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:31.509096 29032 ts_tablet_manager.cc:1888] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:27:31.537521 26476 raft_consensus.cc:1203] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:31.550527 29032 log.cc:1196] T 566929e603d54be4bf931413f0fa1d7d P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:31.552206 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 566929e603d54be4bf931413f0fa1d7d (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
I20240502 21:27:31.558566 29107 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:31.612736 29083 ts_tablet_manager.cc:1888] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:31.613027 29083 log.cc:1196] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/98b5668e0d504db5a7874f041c14dbfa
I20240502 21:27:31.615665 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.619411 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:31.625670 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 98b5668e0d504db5a7874f041c14dbfa (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:31.626443 29083 tablet_replica.cc:330] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:31.626675 29083 raft_consensus.cc:2229] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:31.627115 29083 raft_consensus.cc:2258] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:31.640723 29083 ts_tablet_manager.cc:1875] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:31.659106 29107 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:31.661283 29107 log.cc:1196] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:31.666607 24232 catalog_manager.cc:4830] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): tablet 477d9df7d3a74b8584f3669be3256e00 (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
I20240502 21:27:31.708151 29083 ts_tablet_manager.cc:1888] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:31.712085 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:31.737900 29083 log.cc:1196] T 8f912eb05c574368909e0b3a8edf058d P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f912eb05c574368909e0b3a8edf058d
I20240502 21:27:31.738999 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 8f912eb05c574368909e0b3a8edf058d (table rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]) successfully deleted
I20240502 21:27:31.761425 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.765800 29083 tablet_replica.cc:330] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:31.766166 29083 raft_consensus.cc:2229] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:31.768524 29083 raft_consensus.cc:2258] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:31.779559 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:31.789168 29083 ts_tablet_manager.cc:1875] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:31.807642 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.826486 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.860055 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:31.866817 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:31.900836 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:31.902804 28848 raft_consensus.cc:1050] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:31.904239 28848 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:31.929852 28824 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:31.930385 28824 tablet_bootstrap.cc:492] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:31.930692 28824 ts_tablet_manager.cc:1367] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 1.583s user 0.056s sys 0.009s
I20240502 21:27:31.933738 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.934835 28891 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:31.935322 28891 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:31.935635 28891 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 1.754s user 0.064s sys 0.006s
I20240502 21:27:31.940140 24782 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.940547 26342 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.941395 28524 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.944169 28524 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.951021 26612 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:31.951944 28524 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:31.955078 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:31.970415 28824 raft_consensus.cc:348] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:31.970770 28824 raft_consensus.cc:729] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:31.971065 28824 consensus_queue.cc:260] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:31.971689 28824 ts_tablet_manager.cc:1398] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20240502 21:27:31.974269 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:32.006687 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:32.012291 28891 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:32.012771 28891 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:32.013182 28891 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:32.015214 28891 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.079s user 0.003s sys 0.000s
I20240502 21:27:32.027618 29083 ts_tablet_manager.cc:1888] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:27:32.027801 29083 log.cc:1196] T d0ba1af1d3624d6387bf892873f3f67e P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/d0ba1af1d3624d6387bf892873f3f67e
I20240502 21:27:32.042866 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:32.043282 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet d0ba1af1d3624d6387bf892873f3f67e (table rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]) successfully deleted
I20240502 21:27:32.047767 29083 tablet_replica.cc:330] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:32.055259 29083 raft_consensus.cc:2229] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:32.055908 29083 raft_consensus.cc:2258] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:32.057222 29083 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:32.074685 28845 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.080744 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:32.080912 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-e847a796029d4edea79694f5402a291a received from {username='slave'} at 127.16.122.8:33001
I20240502 21:27:32.079972 24782 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.081063 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-e847a796029d4edea79694f5402a291a on tablet e847a796029d4edea79694f5402a291a with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:32.082576 26342 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.083204 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-bdf23b7a397441b5947ce6babe3e8f62 received from {username='slave'} at 127.16.122.5:41749
I20240502 21:27:32.083352 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session b77f8f52372241d483f457389fda0f93-bdf23b7a397441b5947ce6babe3e8f62 on tablet bdf23b7a397441b5947ce6babe3e8f62 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:32.089483 26612 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.095553 24244 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.177675 29083 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 21:27:32.177850 29083 log.cc:1196] T 477d9df7d3a74b8584f3669be3256e00 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:32.178694 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 477d9df7d3a74b8584f3669be3256e00 (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
I20240502 21:27:32.219182 29172 raft_consensus.cc:1050] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
W20240502 21:27:32.232478 24718 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.6:41751, user_credentials={real_user=slave}} blocked reactor thread for 74527.1us
I20240502 21:27:32.238200 29172 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:32.249063 26342 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.249225 26476 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.249958 29172 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.251286 24522 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.252688 29037 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.253275 29037 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.275094 29037 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.281562 26342 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.289072 24247 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.294431 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:32.295058 26479 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.295396 24522 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.317431 24782 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:32.318879 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:32.322417 26342 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:32.325933 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:32.327072 29174 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 21:27:32.329066 29172 consensus_queue.cc:1159] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7f8bb40905da4891b461220b95c66832" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20240502 21:27:32.330649 26476 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:32.333743 29172 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:32.334842 26342 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.340188 29037 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.347754 26476 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.350772 26344 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:32.351519 28845 consensus_queue.cc:1037] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:32.355358 26741 raft_consensus.cc:1203] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:32.364241 26344 raft_consensus.cc:858] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
W20240502 21:27:32.364414 26344 raft_consensus.cc:1565] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:32.371330 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:32.373971 24247 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:32.374889 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:32.388394 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:32.393733 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:32.415320 24522 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:32.418458 29203 tablet_replica.cc:330] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:32.418865 29203 raft_consensus.cc:2229] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:32.419417 29203 raft_consensus.cc:2258] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:32.429633 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:32.431062 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:32.437424 29203 ts_tablet_manager.cc:1875] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:32.442729 24782 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:32.443974 29004 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:32.451893 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e5a87719c2544a5b0cb591ed9f46df8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.452153 29203 ts_tablet_manager.cc:1888] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:32.474207 26342 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:32.479903 26344 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:32.503530 28848 raft_consensus.cc:1050] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:32.504431 28848 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:32.506806 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:32.524171 26344 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:32.475118 29004 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:32.530511 29203 log.cc:1196] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b2661e0b5fe4a0b94c150b14b8fd9ec
I20240502 21:27:32.532428 29208 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.533807 24782 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.536752 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec (table rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]) successfully deleted
I20240502 21:27:32.547096 26344 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.553145 24247 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.576735 26342 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.611697 28524 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20240502 21:27:32.649886 26476 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.656965 29216 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.666798 24651 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.668062 28848 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.690337 28848 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.697618 24651 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.697795 26342 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:32.699163 24247 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:32.703110 24649 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:32.708097 24522 consensus_queue.cc:237] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:32.708143 26479 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
W20240502 21:27:32.723310 24587 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc5c40e2bf0340958ecd3e7541209e95. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.723779 26742 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:32.724423 24520 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:32.725363 29096 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:32.725879 29096 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:32.734146 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c6e7eba855e48c0b0e18fa363ea01d4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.734365 24651 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:32.735090 29208 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:32.751869 29236 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:32.758090 24520 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:32.759358 26742 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:32.766292 24241 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:32.786164 24782 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:32.786881 29208 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:32.796432 29208 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.799273 24778 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.805176 24649 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.809274 26476 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:32.809756 24389 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:32.810016 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4aa176fd994cf0b55f304cf9d06fe0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:32.810701 29211 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:32.812083 24586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 43540.8us
I20240502 21:27:32.812754 29211 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:32.813040 24520 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:32.825457 24241 catalog_manager.cc:5484] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:32.832716 29210 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:32.839226 26476 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:32.843396 24389 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:32.860561 24247 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:32.883664 29262 ts_tablet_manager.cc:902] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:32.905841 29037 raft_consensus.cc:1050] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:32.907019 29037 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:32.932140 26742 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.933935 29180 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.938383 24520 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:32.940160 26342 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.941028 29180 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.949549 24649 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:32.950363 29176 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:32.965739 24778 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:32.980010 26344 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:32.981199 29255 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:32.983083 24649 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:32.983937 29208 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:32.994946 26742 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:32.995863 29207 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:32.996096 24520 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:33.010512 26612 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:33.011368 26479 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:33.012190 29255 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:33.012957 29207 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:33.039139 26610 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:33.044417 29208 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.045639 29262 tablet_copy_client.cc:308] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:33.047102 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 2e5a87719c2544a5b0cb591ed9f46df8 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:33.047310 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 2e5a87719c2544a5b0cb591ed9f46df8 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-2e5a87719c2544a5b0cb591ed9f46df8
I20240502 21:27:33.037956 29207 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:33.052371 26344 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.055923 24649 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
W20240502 21:27:33.064297 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff7e019fa82d4648afaf130d3f6343d6. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:33.064518 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 76045.8us
W20240502 21:27:33.064821 24456 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.064694 24247 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 11 to 12, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.075348 26742 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.075937 29180 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.081702 24241 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 10 to 11. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.088616 24541 tablet_copy_source_session.cc:205] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.091145 29262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5a87719c2544a5b0cb591ed9f46df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.093724 24651 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.094707 24520 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:33.097616 26742 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:33.098358 29210 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:33.100349 29262 tablet_copy_client.cc:783] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.100684 29262 tablet_copy_client.cc:647] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.102160 29210 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.103665 24247 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.108036 26741 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.116627 26342 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:33.117496 29262 tablet_copy_client.cc:523] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.117529 29255 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:33.119319 26342 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.119966 24247 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 11 to 12, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:33.122509 29262 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:33.123876 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fc717a1b12e4380a4107e6deae2b837 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:33.134302 26479 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:33.140002 24520 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
W20240502 21:27:33.142717 24719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.3:38427, user_credentials={real_user=slave}} blocked reactor thread for 43778.5us
W20240502 21:27:33.145956 24456 consensus_peers.cc:485] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fc717a1b12e4380a4107e6deae2b837. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.146934 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:33.149487 26742 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:33.150641 26342 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:33.150694 29208 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240502 21:27:33.153357 29210 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 21:27:33.154255 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 89246us
I20240502 21:27:33.158092 26344 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.159382 29255 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:33.163952 26741 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:33.162098 26742 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.180938 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0fc717a1b12e4380a4107e6deae2b837 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:33.183579 26343 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:33.188004 24241 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.199389 26342 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:33.200358 29298 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:33.217877 24782 consensus_queue.cc:237] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:33.225224 29307 ts_tablet_manager.cc:902] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:33.225601 26344 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:33.226598 29298 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:33.227999 29304 tablet_replica.cc:330] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:33.229768 29304 raft_consensus.cc:2229] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:33.230109 29304 raft_consensus.cc:2258] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:33.231235 29304 ts_tablet_manager.cc:1875] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:33.233222 29301 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.234553 26345 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.239828 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e847a796029d4edea79694f5402a291a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:33.249183 26739 raft_consensus.cc:1261] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:33.249173 24247 catalog_manager.cc:5484] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.250334 29301 consensus_queue.cc:1035] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:33.254529 29303 ts_tablet_manager.cc:902] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:33.254773 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet e847a796029d4edea79694f5402a291a with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:33.259315 24520 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:33.260115 26742 raft_consensus.cc:2932] T e847a796029d4edea79694f5402a291a P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:33.260849 29302 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:33.287827 29307 tablet_copy_client.cc:308] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:33.288893 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet dc5c40e2bf0340958ecd3e7541209e95 from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:55613)
I20240502 21:27:33.289074 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet dc5c40e2bf0340958ecd3e7541209e95 from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:55613: session id = 9b0952b24bd8400c9940dbd5e28417b4-dc5c40e2bf0340958ecd3e7541209e95
I20240502 21:27:33.292470 24673 tablet_copy_source_session.cc:205] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.298177 29301 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:33.299314 29307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5c40e2bf0340958ecd3e7541209e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.305889 26342 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:33.306162 24242 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.320881 29303 tablet_copy_client.cc:308] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:33.323850 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:33.324193 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 5c6e7eba855e48c0b0e18fa363ea01d4 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-5c6e7eba855e48c0b0e18fa363ea01d4
I20240502 21:27:33.327200 24541 tablet_copy_source_session.cc:205] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.327258 29307 tablet_copy_client.cc:783] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.327666 24778 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:33.327838 29307 tablet_copy_client.cc:647] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.329087 29303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6e7eba855e48c0b0e18fa363ea01d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.342293 29307 tablet_copy_client.cc:523] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.346911 29307 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:33.348359 29303 tablet_copy_client.cc:783] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.348732 29303 tablet_copy_client.cc:647] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.354447 29303 tablet_copy_client.cc:523] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.355073 29262 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.355573 29262 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:33.355868 29262 ts_tablet_manager.cc:1367] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.233s user 0.047s sys 0.012s
I20240502 21:27:33.356732 29262 raft_consensus.cc:348] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:33.357010 29262 raft_consensus.cc:729] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:33.357249 29262 consensus_queue.cc:260] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:33.317404 24520 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:33.367203 26612 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 21:27:33.367460 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.368052 29302 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:33.369406 24522 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:33.370033 29302 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:33.375187 29262 ts_tablet_manager.cc:1398] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20240502 21:27:33.378000 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-2e5a87719c2544a5b0cb591ed9f46df8 received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:33.378257 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-2e5a87719c2544a5b0cb591ed9f46df8 on tablet 2e5a87719c2544a5b0cb591ed9f46df8 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:33.385257 29301 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:33.386209 26612 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:33.389616 24519 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:33.393258 29303 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:33.394785 26479 raft_consensus.cc:1203] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:33.394845 24241 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 8 to 9, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.398696 29304 ts_tablet_manager.cc:1888] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 21:27:33.398854 29304 log.cc:1196] T 0fc717a1b12e4380a4107e6deae2b837 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:33.427670 29342 ts_tablet_manager.cc:902] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:33.433292 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 0fc717a1b12e4380a4107e6deae2b837 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:33.437165 29304 tablet_replica.cc:330] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:33.437531 29304 raft_consensus.cc:2229] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:33.437777 29345 ts_tablet_manager.cc:902] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:33.438028 29304 raft_consensus.cc:2258] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:33.439198 29304 ts_tablet_manager.cc:1875] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:33.485157 29304 ts_tablet_manager.cc:1888] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:33.485693 29304 log.cc:1196] T e847a796029d4edea79694f5402a291a P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/e847a796029d4edea79694f5402a291a
I20240502 21:27:33.488302 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet e847a796029d4edea79694f5402a291a (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:33.489173 24520 consensus_queue.cc:237] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
W20240502 21:27:33.494860 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ee3718b120e4fad9ae03e47d3f1fffa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:33.497316 24649 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:33.498152 29210 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:33.501549 24389 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:33.502388 29210 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:33.512683 29255 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:33.518585 24389 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:33.531404 24242 catalog_manager.cc:5484] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.546871 29345 tablet_copy_client.cc:308] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:33.555338 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet ff7e019fa82d4648afaf130d3f6343d6 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:33001)
I20240502 21:27:33.555546 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet ff7e019fa82d4648afaf130d3f6343d6 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:33001: session id = 1769ee000eb24298a367a0de1209e6e0-ff7e019fa82d4648afaf130d3f6343d6
I20240502 21:27:33.559679 24804 tablet_copy_source_session.cc:205] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.560631 29307 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.548245 29342 tablet_copy_client.cc:308] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:33.561194 29307 tablet_bootstrap.cc:492] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:33.561594 29307 ts_tablet_manager.cc:1367] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.215s user 0.053s sys 0.020s
I20240502 21:27:33.561774 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:34905)
I20240502 21:27:33.561955 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 6c4aa176fd994cf0b55f304cf9d06fe0 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:34905: session id = 1769ee000eb24298a367a0de1209e6e0-6c4aa176fd994cf0b55f304cf9d06fe0
I20240502 21:27:33.562696 29307 raft_consensus.cc:348] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:33.562992 29307 raft_consensus.cc:729] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:33.563203 29307 consensus_queue.cc:260] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:33.563840 29307 ts_tablet_manager.cc:1398] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:33.564587 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-dc5c40e2bf0340958ecd3e7541209e95 received from {username='slave'} at 127.16.122.7:55613
I20240502 21:27:33.564752 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-dc5c40e2bf0340958ecd3e7541209e95 on tablet dc5c40e2bf0340958ecd3e7541209e95 with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:33.566605 24541 tablet_copy_source_session.cc:205] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.569893 29345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7e019fa82d4648afaf130d3f6343d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.570959 29342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4aa176fd994cf0b55f304cf9d06fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.579939 29378 ts_tablet_manager.cc:902] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:33.601464 24649 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:33.604801 26612 raft_consensus.cc:1203] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:33.611474 29307 ts_tablet_manager.cc:902] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:33.615969 29345 tablet_copy_client.cc:783] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.617237 29345 tablet_copy_client.cc:647] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.621344 29345 tablet_copy_client.cc:523] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.629693 29345 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:33.635960 29342 tablet_copy_client.cc:783] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.655074 29342 tablet_copy_client.cc:647] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.657302 29342 tablet_copy_client.cc:523] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.670912 29378 tablet_copy_client.cc:308] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:33.673022 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet ac35b080037646b4b8b34c0eab2de4e6 from peer 1398472b92c842f285b8c7cc43cc1527 ({username='slave'} at 127.16.122.4:54005)
I20240502 21:27:33.673295 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet ac35b080037646b4b8b34c0eab2de4e6 from peer 1398472b92c842f285b8c7cc43cc1527 at {username='slave'} at 127.16.122.4:54005: session id = 1398472b92c842f285b8c7cc43cc1527-ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:33.677273 24541 tablet_copy_source_session.cc:205] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.679895 29378 ts_tablet_manager.cc:1875] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:33.682252 29342 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:33.690534 29307 tablet_copy_client.cc:308] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:33.691314 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 0fc717a1b12e4380a4107e6deae2b837 from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:35031)
I20240502 21:27:33.691594 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 0fc717a1b12e4380a4107e6deae2b837 from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:35031: session id = 9b0952b24bd8400c9940dbd5e28417b4-0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:33.694967 24541 tablet_copy_source_session.cc:205] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.697332 29307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc717a1b12e4380a4107e6deae2b837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.722309 29307 tablet_copy_client.cc:783] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.722823 29307 tablet_copy_client.cc:647] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.725234 29307 tablet_copy_client.cc:523] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.733086 29307 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:33.735181 29378 ts_tablet_manager.cc:1888] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240502 21:27:33.735538 29378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac35b080037646b4b8b34c0eab2de4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.809854 29378 tablet_copy_client.cc:783] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.810230 29378 tablet_copy_client.cc:647] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.812482 29378 tablet_copy_client.cc:523] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.827630 29421 ts_tablet_manager.cc:902] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:33.844789 29378 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:33.859076 29421 tablet_copy_client.cc:308] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:33.860067 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:33001)
I20240502 21:27:33.860348 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:33001: session id = 1769ee000eb24298a367a0de1209e6e0-7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:33.866516 24804 tablet_copy_source_session.cc:205] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:33.869423 29421 ts_tablet_manager.cc:1875] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:33.925258 29421 ts_tablet_manager.cc:1888] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 21:27:33.925633 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.934604 29303 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.935302 29303 tablet_bootstrap.cc:492] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:33.935967 29307 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.938439 29421 tablet_copy_client.cc:783] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:33.938774 29421 tablet_copy_client.cc:647] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:33.941191 29303 ts_tablet_manager.cc:1367] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.548s user 0.061s sys 0.008s
I20240502 21:27:33.944449 29303 raft_consensus.cc:348] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:33.944744 29303 raft_consensus.cc:729] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:33.944969 29303 consensus_queue.cc:260] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:33.945780 29421 tablet_copy_client.cc:523] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:33.956287 29463 ts_tablet_manager.cc:902] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:33.957088 29303 ts_tablet_manager.cc:1398] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20240502 21:27:33.957850 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-5c6e7eba855e48c0b0e18fa363ea01d4 received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:33.958003 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-5c6e7eba855e48c0b0e18fa363ea01d4 on tablet 5c6e7eba855e48c0b0e18fa363ea01d4 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:33.966574 29307 tablet_bootstrap.cc:492] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:33.966956 29421 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:33.966935 29307 ts_tablet_manager.cc:1367] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 0.234s user 0.067s sys 0.005s
I20240502 21:27:33.973120 29345 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.973640 29345 tablet_bootstrap.cc:492] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:33.975953 29307 raft_consensus.cc:348] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:33.976226 29307 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:33.976512 29307 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:33.984066 29342 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:33.984760 29342 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:33.985118 29342 ts_tablet_manager.cc:1367] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.303s user 0.051s sys 0.012s
I20240502 21:27:33.986397 29342 raft_consensus.cc:348] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:33.986743 29342 raft_consensus.cc:729] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:33.987023 29342 consensus_queue.cc:260] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:33.994097 29255 raft_consensus.cc:1050] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:33.994053 29307 ts_tablet_manager.cc:1398] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 21:27:33.995306 29255 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:34.002081 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-0fc717a1b12e4380a4107e6deae2b837 received from {username='slave'} at 127.16.122.7:35031
I20240502 21:27:34.002275 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-0fc717a1b12e4380a4107e6deae2b837 on tablet 0fc717a1b12e4380a4107e6deae2b837 with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:34.003232 29345 ts_tablet_manager.cc:1367] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.373s user 0.055s sys 0.006s
I20240502 21:27:34.004453 29342 ts_tablet_manager.cc:1398] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.019s user 0.000s sys 0.011s
I20240502 21:27:34.005869 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-6c4aa176fd994cf0b55f304cf9d06fe0 received from {username='slave'} at 127.16.122.8:34905
I20240502 21:27:34.006023 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-6c4aa176fd994cf0b55f304cf9d06fe0 on tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:34.009886 29345 raft_consensus.cc:348] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:34.010273 29345 raft_consensus.cc:729] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:34.010612 29345 consensus_queue.cc:260] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:34.011444 29345 ts_tablet_manager.cc:1398] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:34.012344 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-ff7e019fa82d4648afaf130d3f6343d6 received from {username='slave'} at 127.16.122.8:33001
I20240502 21:27:34.012526 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-ff7e019fa82d4648afaf130d3f6343d6 on tablet ff7e019fa82d4648afaf130d3f6343d6 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:34.024430 26479 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.024710 26342 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.024786 29463 tablet_copy_client.cc:308] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:34.025171 29208 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.026187 29210 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.029340 24778 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.031280 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 2ee3718b120e4fad9ae03e47d3f1fffa from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:34445)
I20240502 21:27:34.031533 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 2ee3718b120e4fad9ae03e47d3f1fffa from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:34445: session id = 836775e329f148e5908a1afc54f06428-2ee3718b120e4fad9ae03e47d3f1fffa
I20240502 21:27:34.034852 29210 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.049827 24541 tablet_copy_source_session.cc:205] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:34.052889 26476 raft_consensus.cc:1203] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:34.053442 29463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee3718b120e4fad9ae03e47d3f1fffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.089037 29208 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.097649 26342 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.100057 26479 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.111382 24242 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:34.125478 26612 raft_consensus.cc:1203] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240502 21:27:34.128859 24778 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.174772 29096 raft_consensus.cc:1050] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:34.175678 29096 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:34.198040 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.213387 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:34.233805 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:34.245083 26742 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.245893 29505 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.267494 29463 tablet_copy_client.cc:783] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:34.268023 29463 tablet_copy_client.cc:647] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:34.271113 29463 tablet_copy_client.cc:523] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:34.286444 24520 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.286847 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:34.287165 29505 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.305917 26612 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.306668 29096 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.308279 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.308990 29421 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.309458 29421 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:34.309783 29421 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.343s user 0.061s sys 0.004s
I20240502 21:27:34.313388 26739 raft_consensus.cc:1203] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:34.320300 24519 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:34.326426 26479 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:34.327121 29479 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:34.328349 29421 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:34.328677 29421 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:34.328891 29421 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:34.329615 29421 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240502 21:27:34.330435 29479 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.332199 26342 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:34.332824 29210 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:34.333493 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-7a35079a106c4cca94cf65ffa93aefbf received from {username='slave'} at 127.16.122.8:33001
I20240502 21:27:34.333637 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-7a35079a106c4cca94cf65ffa93aefbf on tablet 7a35079a106c4cca94cf65ffa93aefbf with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:34.334389 26479 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.334975 26342 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.336261 29463 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:34.343142 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e5a87719c2544a5b0cb591ed9f46df8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:34.346633 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.361400 26739 raft_consensus.cc:1203] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 21:27:34.374481 29505 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.382026 24520 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.393273 24244 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 8 to 9, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.404533 26742 raft_consensus.cc:1203] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:34.446159 24248 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.448582 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5a87719c2544a5b0cb591ed9f46df8 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:34.449748 26742 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.491422 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:34.501300 29545 tablet_replica.cc:330] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:34.503122 29545 raft_consensus.cc:2229] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:34.504197 29545 raft_consensus.cc:2258] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:34.506103 29545 ts_tablet_manager.cc:1875] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:34.519397 26612 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.560676 29545 ts_tablet_manager.cc:1888] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:34.562639 29545 log.cc:1196] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e5a87719c2544a5b0cb591ed9f46df8
I20240502 21:27:34.564297 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet 2e5a87719c2544a5b0cb591ed9f46df8 (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:34.588182 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.588871 29494 raft_consensus.cc:1050] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:34.589848 29494 consensus_queue.cc:237] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:34.608263 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:34.625919 26479 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:34.626843 29208 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:34.634449 24778 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:34.635260 29208 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:34.649689 29210 raft_consensus.cc:1050] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:34.650671 29210 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:34.666545 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.668653 26342 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:34.669620 29479 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:34.674618 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.677322 26739 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:34.678061 29208 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:34.689484 26612 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:34.690310 29210 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:34.702641 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:34.710744 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:34.712899 29208 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.718515 26477 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.719234 24782 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.729171 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:34.736833 24246 catalog_manager.cc:5484] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.740593 29378 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:34.741711 29378 tablet_bootstrap.cc:492] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap complete.
I20240502 21:27:34.742710 29378 ts_tablet_manager.cc:1367] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.898s user 0.066s sys 0.011s
I20240502 21:27:34.742729 29255 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.744158 29378 raft_consensus.cc:348] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:34.744463 29378 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: LEARNER
I20240502 21:27:34.744690 29378 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:34.745397 29378 ts_tablet_manager.cc:1398] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 21:27:34.745972 24520 consensus_queue.cc:237] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:34.748214 26342 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.749867 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.751922 26476 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:34.754550 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 1398472b92c842f285b8c7cc43cc1527-ac35b080037646b4b8b34c0eab2de4e6 received from {username='slave'} at 127.16.122.4:54005
I20240502 21:27:34.754683 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 1398472b92c842f285b8c7cc43cc1527-ac35b080037646b4b8b34c0eab2de4e6 on tablet ac35b080037646b4b8b34c0eab2de4e6 with peer 1398472b92c842f285b8c7cc43cc1527
I20240502 21:27:34.754921 29255 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:34.758423 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.758265 24242 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.762637 24782 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:34.763727 29255 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:34.765583 26610 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.768880 29494 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.793486 26477 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.793764 26739 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:34.830957 24247 catalog_manager.cc:5484] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:34.842188 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.869601 29255 raft_consensus.cc:1050] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:34.870568 29255 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:34.874558 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:34.881369 24389 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.881898 26477 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.882055 29479 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.883419 29479 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.886809 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.890285 26742 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:34.891623 29210 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:34.893263 24649 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:34.894752 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:34.924801 24631 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:34.929921 29299 raft_consensus.cc:1050] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:34.931082 29299 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:34.946060 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:34.952152 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:34.956221 29479 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:34.970435 26477 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:34.979446 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.980743 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:34.993957 24389 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:34.996759 29608 tablet_replica.cc:330] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:34.996999 29608 raft_consensus.cc:2229] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:35.006881 29608 raft_consensus.cc:2258] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:35.007730 29608 ts_tablet_manager.cc:1875] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:35.029268 24782 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:35.045197 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:35.047874 24242 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.077454 26610 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:35.078497 29618 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:35.078406 26742 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.079794 26739 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:35.087888 24782 raft_consensus.cc:1203] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240502 21:27:35.094882 29617 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 21:27:35.081153 29621 raft_consensus.cc:1050] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:35.097254 29621 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:35.115855 26740 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:35.116829 26344 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:35.117039 24520 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:35.120375 29618 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:35.121402 29617 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:35.129536 29618 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:35.155059 24519 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:35.155915 29299 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:35.164013 26739 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.168336 29621 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.171648 24242 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.183719 29620 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.189719 24520 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.190743 26739 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.197880 24247 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 9 to 10, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:35.212009 24520 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.220322 26344 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.255420 24782 consensus_queue.cc:237] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:35.258714 26344 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.258744 26739 raft_consensus.cc:1261] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.259500 29616 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.260514 29616 consensus_queue.cc:1035] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.263744 29617 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.271263 24240 catalog_manager.cc:5484] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:35.273536 26610 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.274245 26344 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.276233 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet ff7e019fa82d4648afaf130d3f6343d6 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:35.286000 26739 raft_consensus.cc:2932] T ff7e019fa82d4648afaf130d3f6343d6 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.297931 29637 tablet_replica.cc:330] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:35.304302 29637 raft_consensus.cc:2229] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:35.304827 29637 raft_consensus.cc:2258] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:35.307535 29637 ts_tablet_manager.cc:1875] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:35.312227 24782 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:35.314337 26610 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:35.315001 29616 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:35.316339 26739 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:35.317061 29616 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:35.318471 29616 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.322779 26610 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.330950 29637 ts_tablet_manager.cc:1888] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:35.331212 29637 log.cc:1196] T ff7e019fa82d4648afaf130d3f6343d6 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff7e019fa82d4648afaf130d3f6343d6
I20240502 21:27:35.331681 26741 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.332477 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet ff7e019fa82d4648afaf130d3f6343d6 (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
I20240502 21:27:35.333149 24240 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 10 to 11, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.339407 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:35.343639 29637 tablet_replica.cc:330] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:35.343923 29637 raft_consensus.cc:2229] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:35.344514 29637 raft_consensus.cc:2258] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:35.345949 29637 ts_tablet_manager.cc:1875] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:35.350996 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff7e019fa82d4648afaf130d3f6343d6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:35.351302 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a35079a106c4cca94cf65ffa93aefbf with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:35.355988 24520 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:35.356663 29637 ts_tablet_manager.cc:1888] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 21:27:35.358191 24389 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.359936 29255 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.360457 26739 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.361228 29255 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.356879 29637 log.cc:1196] T 7a35079a106c4cca94cf65ffa93aefbf P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:35.365362 29255 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.367295 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 7a35079a106c4cca94cf65ffa93aefbf (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:35.368193 24389 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:35.369503 24384 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.380273 26741 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.391176 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:35.402221 26476 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.402446 26344 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.403182 29224 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.403985 29220 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.405727 24242 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 8 to 9, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:35.403045 24648 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:35.407019 29220 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:35.379164 29608 ts_tablet_manager.cc:1888] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:27:35.422371 29608 log.cc:1196] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c6e7eba855e48c0b0e18fa363ea01d4
I20240502 21:27:35.423542 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 5c6e7eba855e48c0b0e18fa363ea01d4 (table rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]) successfully deleted
I20240502 21:27:35.429229 29220 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:35.426537 26458 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with delete_type TABLET_DATA_TOMBSTONED (TS 836775e329f148e5908a1afc54f06428 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51488
I20240502 21:27:35.431186 26344 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:35.443028 29463 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:35.443722 29463 tablet_bootstrap.cc:492] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:35.444458 29463 ts_tablet_manager.cc:1367] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 1.108s user 0.054s sys 0.011s
I20240502 21:27:35.446154 29463 raft_consensus.cc:348] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:35.446870 29463 raft_consensus.cc:729] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:35.447238 29463 consensus_queue.cc:260] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:35.447405 24240 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:35.440651 24648 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:35.461619 29463 ts_tablet_manager.cc:1398] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240502 21:27:35.461508 26477 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:35.471334 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-2ee3718b120e4fad9ae03e47d3f1fffa received from {username='slave'} at 127.16.122.6:34445
I20240502 21:27:35.471498 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 836775e329f148e5908a1afc54f06428-2ee3718b120e4fad9ae03e47d3f1fffa on tablet 2ee3718b120e4fad9ae03e47d3f1fffa with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:35.473145 24389 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:35.476275 26344 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:35.476893 29224 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 21:27:35.478332 24323 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:35.482600 29648 tablet_replica.cc:330] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: stopping tablet replica
I20240502 21:27:35.482978 29648 raft_consensus.cc:2229] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:35.486186 26479 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:35.487053 29648 raft_consensus.cc:2258] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:35.491142 29648 ts_tablet_manager.cc:1875] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:35.493142 24649 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:35.494522 29224 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:35.495244 29224 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:35.500535 29224 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.503535 24649 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.512542 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bdf23b7a397441b5947ce6babe3e8f62 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:35.513571 26344 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.518458 24240 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.505996 26477 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.540395 24389 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:35.544910 26344 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:35.545589 29599 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:35.546797 26477 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:35.547665 29655 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:35.550370 29599 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.552640 26476 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.554116 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:35.557035 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdf23b7a397441b5947ce6babe3e8f62 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:35.557154 26344 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:35.559298 24247 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 10 to 11, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.566444 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf23b7a397441b5947ce6babe3e8f62 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:35.566875 29608 tablet_replica.cc:330] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:35.567099 29608 raft_consensus.cc:2229] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:35.567596 29608 raft_consensus.cc:2258] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:35.569044 29608 ts_tablet_manager.cc:1875] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:35.584942 29648 ts_tablet_manager.cc:1888] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:35.586273 29648 log.cc:1196] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 836775e329f148e5908a1afc54f06428: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c4aa176fd994cf0b55f304cf9d06fe0
I20240502 21:27:35.588634 24233 catalog_manager.cc:4830] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): tablet 6c4aa176fd994cf0b55f304cf9d06fe0 (table rebalance_test_table_2 [id=8a90449876554e7393730fffdb194d49]) successfully deleted
I20240502 21:27:35.597261 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.605042 29608 ts_tablet_manager.cc:1888] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 21:27:35.605484 29608 log.cc:1196] T bdf23b7a397441b5947ce6babe3e8f62 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/bdf23b7a397441b5947ce6babe3e8f62
W20240502 21:27:35.609699 24323 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:35.616947 24782 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:35.618273 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet bdf23b7a397441b5947ce6babe3e8f62 (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
I20240502 21:27:35.621002 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:35.631673 29479 raft_consensus.cc:1050] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 1398472b92c842f285b8c7cc43cc1527 to VOTER
I20240502 21:27:35.632658 29479 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
W20240502 21:27:35.640506 24717 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:35.641865 26477 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:35.642696 29616 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:35.647153 24649 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:35.647884 29616 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:35.656416 24782 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:35.657045 29255 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:35.657148 26610 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:35.657933 29494 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:35.668426 26477 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:35.669211 29479 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:35.675704 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:35.683257 29616 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:35.691545 26477 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:35.695037 24240 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.702042 29255 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:35.703208 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.707527 24778 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:35.708804 26610 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:35.709434 26477 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:35.715170 24247 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 12 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:35.722326 24649 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:35.767932 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.789714 26324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:35.867504 24389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62"
dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34660
I20240502 21:27:35.867830 24389 raft_consensus.cc:595] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:35.879895 24370 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:35.881053 26477 raft_consensus.cc:1203] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:35.907333 24782 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:35.911959 26741 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:35.912729 29616 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 21:27:35.913255 24719 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): Couldn't send request to peer 427345e9a65d417c84a773ac1925a18a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:35.914105 24520 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:35.916045 29618 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:35.916939 26741 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:35.926452 26610 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:35.927260 29621 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:35.929757 26610 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:35.951572 24240 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 11 to 12, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.970925 29224 raft_consensus.cc:979] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: : Instructing follower b77f8f52372241d483f457389fda0f93 to start an election
I20240502 21:27:35.971150 29224 raft_consensus.cc:1067] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Signalling peer b77f8f52372241d483f457389fda0f93 to start an election
I20240502 21:27:35.972160 26344 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62"
dest_uuid: "b77f8f52372241d483f457389fda0f93"
from {username='slave'} at 127.16.122.1:50813
I20240502 21:27:35.972446 26344 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:35.972659 26344 raft_consensus.cc:3037] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:35.977735 26344 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:35.981683 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62" candidate_uuid: "b77f8f52372241d483f457389fda0f93" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:35.981969 24389 raft_consensus.cc:3032] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:35.982676 24389 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:35.983245 24389 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:35.983916 24389 raft_consensus.cc:3037] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:35.987749 24389 raft_consensus.cc:2454] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77f8f52372241d483f457389fda0f93 in term 2.
I20240502 21:27:35.988276 26344 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [CANDIDATE]: Term 2 election: Requested vote from peers e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751)
I20240502 21:27:35.990072 26277 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b77f8f52372241d483f457389fda0f93, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:35.990721 29506 raft_consensus.cc:2781] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:36.003922 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdf23b7a397441b5947ce6babe3e8f62" candidate_uuid: "b77f8f52372241d483f457389fda0f93" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "836775e329f148e5908a1afc54f06428"
I20240502 21:27:36.004364 26477 raft_consensus.cc:3037] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:36.008127 26477 raft_consensus.cc:2454] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77f8f52372241d483f457389fda0f93 in term 2.
I20240502 21:27:36.009845 29506 raft_consensus.cc:686] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 2 LEADER]: Becoming Leader. State: Replica: b77f8f52372241d483f457389fda0f93, State: Running, Role: LEADER
W20240502 21:27:36.012156 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f912eb05c574368909e0b3a8edf058d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.013556 26610 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:36.014266 29255 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:36.016163 24389 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:36.017318 29479 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:36.017182 29506 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:36.037940 24247 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 reported cstate change: term changed from 1 to 2, leader changed from e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) to b77f8f52372241d483f457389fda0f93 (127.16.122.5). New cstate: current_term: 2 leader_uuid: "b77f8f52372241d483f457389fda0f93" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:36.081516 29479 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:36.088356 24384 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:36.096642 24242 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:36.111775 29724 ts_tablet_manager.cc:902] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:36.114853 26610 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:36.165813 24649 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:36.166772 29724 tablet_copy_client.cc:308] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:36.168676 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:58509)
I20240502 21:27:36.168916 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:58509: session id = 7f8bb40905da4891b461220b95c66832-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0
I20240502 21:27:36.173271 24804 tablet_copy_source_session.cc:205] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:36.175194 29724 ts_tablet_manager.cc:1875] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:36.186995 29724 ts_tablet_manager.cc:1888] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:27:36.187358 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:36.192839 26344 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:36.194061 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82fb425a5430426f8030ba5497736b11. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.194991 29547 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:36.199390 24520 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:36.200222 29547 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:36.213857 29754 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.218315 29724 tablet_copy_client.cc:783] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:36.218765 29724 tablet_copy_client.cc:647] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:36.235894 29724 tablet_copy_client.cc:523] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:36.239274 26344 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.247004 24242 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.257263 24520 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.265640 29724 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:36.313424 24520 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:36.334044 29255 raft_consensus.cc:1050] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:36.334973 29255 consensus_queue.cc:237] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
W20240502 21:27:36.337173 24457 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): Couldn't send request to peer 427345e9a65d417c84a773ac1925a18a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.338938 24649 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:36.339630 29688 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:36.342208 24389 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:36.343111 26477 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:36.343904 29688 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:36.344480 29688 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:36.376048 26476 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:36.376786 29688 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:36.379995 26344 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:36.380722 29255 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:36.383678 29479 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.391319 24648 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.392434 24651 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:36.392692 26477 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.396502 24242 catalog_manager.cc:5484] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:36.397001 29724 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:36.397531 29724 tablet_bootstrap.cc:492] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:27:36.397959 29724 ts_tablet_manager.cc:1367] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.132s user 0.051s sys 0.014s
I20240502 21:27:36.398912 29724 raft_consensus.cc:348] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:36.399209 29724 raft_consensus.cc:729] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:27:36.399464 29724 consensus_queue.cc:260] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:36.400732 29724 ts_tablet_manager.cc:1398] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:36.402693 24384 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:27:36.403347 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 received from {username='slave'} at 127.16.122.2:58509
I20240502 21:27:36.403509 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 7f8bb40905da4891b461220b95c66832-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 on tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:27:36.403434 29779 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.405704 29506 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:36.407254 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:36.407696 26610 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:36.408154 29547 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:36.409675 24389 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.410696 29754 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:36.412245 26342 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.414503 24520 consensus_queue.cc:237] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:36.419144 24389 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:36.420058 29479 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:36.428333 24240 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.432413 26477 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:36.433444 29779 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:36.438730 26476 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.446450 29255 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.447448 26476 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.453271 26344 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:36.454270 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ee3718b120e4fad9ae03e47d3f1fffa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:36.454741 29754 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:36.467567 24242 catalog_manager.cc:5484] T 2ee3718b120e4fad9ae03e47d3f1fffa P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:36.467998 24386 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
W20240502 21:27:36.474584 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.476380 26476 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:27:36.477399 29506 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:36.487298 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ee3718b120e4fad9ae03e47d3f1fffa with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:36.490866 29547 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:36.499733 29806 ts_tablet_manager.cc:902] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:36.500694 26612 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:36.507964 24242 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 13 to 14. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:36.510239 26344 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:36.516041 24649 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:36.517697 29819 ts_tablet_manager.cc:902] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:36.518990 29816 tablet_replica.cc:330] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:36.519773 29816 raft_consensus.cc:2229] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:36.520416 29816 raft_consensus.cc:2258] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:36.522629 26610 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 21:27:36.523403 29754 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 21:27:36.525153 24586 consensus_peers.cc:485] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566929e603d54be4bf931413f0fa1d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.531607 26739 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 21:27:36.532588 29754 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:36.534329 26741 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:36.534955 29816 ts_tablet_manager.cc:1875] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:36.549207 26345 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 21:27:36.550858 29754 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:36.562530 29754 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.566002 24520 raft_consensus.cc:1203] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:36.567031 29819 tablet_copy_client.cc:308] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:27:36.568351 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 427345e9a65d417c84a773ac1925a18a ({username='slave'} at 127.16.122.3:33699)
I20240502 21:27:36.568537 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7a35079a106c4cca94cf65ffa93aefbf from peer 427345e9a65d417c84a773ac1925a18a at {username='slave'} at 127.16.122.3:33699: session id = 427345e9a65d417c84a773ac1925a18a-7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:36.571643 24804 tablet_copy_source_session.cc:205] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:36.572611 29816 ts_tablet_manager.cc:1888] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:36.572774 29816 log.cc:1196] T 2ee3718b120e4fad9ae03e47d3f1fffa P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/2ee3718b120e4fad9ae03e47d3f1fffa
I20240502 21:27:36.573243 29819 ts_tablet_manager.cc:1875] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:36.575601 24234 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:36.575147 26612 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.575779 29806 tablet_copy_client.cc:308] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:36.576400 24247 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:36.577500 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 8f912eb05c574368909e0b3a8edf058d from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:51181)
I20240502 21:27:36.577661 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 8f912eb05c574368909e0b3a8edf058d from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:51181: session id = b77f8f52372241d483f457389fda0f93-8f912eb05c574368909e0b3a8edf058d
I20240502 21:27:36.580646 24541 tablet_copy_source_session.cc:205] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:36.583060 26344 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.584374 29806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f912eb05c574368909e0b3a8edf058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:36.587424 24649 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:36.591882 26739 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 21:27:36.592185 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 2ee3718b120e4fad9ae03e47d3f1fffa (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
W20240502 21:27:36.592694 24586 consensus_peers.cc:485] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566929e603d54be4bf931413f0fa1d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.592784 26612 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 21:27:36.593060 29547 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:36.593400 29756 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 21:27:36.592710 29819 ts_tablet_manager.cc:1888] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 21:27:36.594051 29819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:36.594702 26739 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.597899 26610 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.596349 29756 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.606331 24242 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 15 to 16, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.606330 26739 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:36.611066 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:36.624588 26324 tablet_service.cc:1514] Processing DeleteTablet for tablet 566929e603d54be4bf931413f0fa1d7d with delete_type TABLET_DATA_TOMBSTONED (TS b77f8f52372241d483f457389fda0f93 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:38914
I20240502 21:27:36.642490 29819 tablet_copy_client.cc:783] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:36.643210 29819 tablet_copy_client.cc:647] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:36.655622 29806 tablet_copy_client.cc:783] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:36.655936 29806 tablet_copy_client.cc:647] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:36.657133 29853 tablet_replica.cc:330] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: stopping tablet replica
I20240502 21:27:36.658532 29853 raft_consensus.cc:2229] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:36.660189 29853 raft_consensus.cc:2258] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:36.661428 29853 ts_tablet_manager.cc:1875] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:36.664639 29819 tablet_copy_client.cc:523] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:36.663198 29806 tablet_copy_client.cc:523] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:36.674984 29819 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:36.679805 29860 ts_tablet_manager.cc:902] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:36.683229 29806 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:36.683554 24386 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:36.699685 29853 ts_tablet_manager.cc:1888] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20240502 21:27:36.699868 29853 log.cc:1196] T 566929e603d54be4bf931413f0fa1d7d P b77f8f52372241d483f457389fda0f93: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/wals/566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:36.700832 24232 catalog_manager.cc:4830] TS b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): tablet 566929e603d54be4bf931413f0fa1d7d (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
W20240502 21:27:36.710937 24324 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): Couldn't send request to peer 427345e9a65d417c84a773ac1925a18a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.712033 26476 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:36.713131 29655 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:36.721830 26610 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:36.722469 29655 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:36.730497 29224 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.739850 26610 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.740564 24240 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:36.754107 26479 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:36.756202 29885 ts_tablet_manager.cc:902] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:36.763636 29860 tablet_copy_client.cc:308] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:36.764438 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 82fb425a5430426f8030ba5497736b11 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:38365)
I20240502 21:27:36.764689 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 82fb425a5430426f8030ba5497736b11 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:38365: session id = 1769ee000eb24298a367a0de1209e6e0-82fb425a5430426f8030ba5497736b11
I20240502 21:27:36.772004 24673 tablet_copy_source_session.cc:205] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:36.773675 29860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb425a5430426f8030ba5497736b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:36.781554 29860 tablet_copy_client.cc:783] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:36.781870 29860 tablet_copy_client.cc:647] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:36.784308 29860 tablet_copy_client.cc:523] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:36.787776 29860 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:36.796936 29885 tablet_copy_client.cc:308] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:27:36.801292 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 9963dcea478749e0a6e076fac6537fc4 from peer 427345e9a65d417c84a773ac1925a18a ({username='slave'} at 127.16.122.3:58123)
I20240502 21:27:36.801578 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 9963dcea478749e0a6e076fac6537fc4 from peer 427345e9a65d417c84a773ac1925a18a at {username='slave'} at 127.16.122.3:58123: session id = 427345e9a65d417c84a773ac1925a18a-9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:36.805768 24541 tablet_copy_source_session.cc:205] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:36.807040 29885 ts_tablet_manager.cc:1875] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:36.826444 29885 ts_tablet_manager.cc:1888] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 21:27:36.826985 29885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9963dcea478749e0a6e076fac6537fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:36.843145 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:36.854310 29885 tablet_copy_client.cc:783] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:36.854936 29885 tablet_copy_client.cc:647] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:36.859756 29860 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:36.860227 29860 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:36.860728 29885 tablet_copy_client.cc:523] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:36.864533 29885 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:36.866698 29860 ts_tablet_manager.cc:1367] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.079s user 0.054s sys 0.012s
W20240502 21:27:36.867777 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:36.869047 26610 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:36.869258 24782 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:36.869801 29655 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:36.868893 29860 raft_consensus.cc:348] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:36.870047 29860 raft_consensus.cc:729] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:36.870388 29655 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:36.870271 29860 consensus_queue.cc:260] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:36.875691 29860 ts_tablet_manager.cc:1398] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:36.876870 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-82fb425a5430426f8030ba5497736b11 received from {username='slave'} at 127.16.122.8:38365
I20240502 21:27:36.877039 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-82fb425a5430426f8030ba5497736b11 on tablet 82fb425a5430426f8030ba5497736b11 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:36.884613 29224 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.889828 24778 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.897478 24247 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.909469 26610 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:27:36.934676 29806 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:36.935251 29806 tablet_bootstrap.cc:492] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:36.935612 29806 ts_tablet_manager.cc:1367] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.265s user 0.058s sys 0.005s
I20240502 21:27:36.939347 29806 raft_consensus.cc:348] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:36.939666 29806 raft_consensus.cc:729] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:36.941280 29806 consensus_queue.cc:260] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:36.945732 29806 ts_tablet_manager.cc:1398] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 21:27:36.949064 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-8f912eb05c574368909e0b3a8edf058d received from {username='slave'} at 127.16.122.5:51181
I20240502 21:27:36.949209 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session b77f8f52372241d483f457389fda0f93-8f912eb05c574368909e0b3a8edf058d on tablet 8f912eb05c574368909e0b3a8edf058d with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:37.085610 29819 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:37.086184 29819 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Bootstrap complete.
I20240502 21:27:37.093394 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:37.110522 26739 raft_consensus.cc:1203] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:37.114738 29621 raft_consensus.cc:1050] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 7f8bb40905da4891b461220b95c66832 to VOTER
I20240502 21:27:37.116068 29621 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:37.119283 29819 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.444s user 0.071s sys 0.004s
I20240502 21:27:37.145056 29819 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:37.145411 29819 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: LEARNER
I20240502 21:27:37.145799 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:37.145681 29819 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:37.162240 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:37.166028 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:37.191277 29966 ts_tablet_manager.cc:902] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:37.192394 29819 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.073s user 0.001s sys 0.001s
I20240502 21:27:37.193928 29965 ts_tablet_manager.cc:902] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:37.208043 29962 ts_tablet_manager.cc:902] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Initiating tablet copy from peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:27:37.211288 26324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:37.214288 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session 427345e9a65d417c84a773ac1925a18a-7a35079a106c4cca94cf65ffa93aefbf received from {username='slave'} at 127.16.122.3:33699
I20240502 21:27:37.214464 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session 427345e9a65d417c84a773ac1925a18a-7a35079a106c4cca94cf65ffa93aefbf on tablet 7a35079a106c4cca94cf65ffa93aefbf with peer 427345e9a65d417c84a773ac1925a18a
I20240502 21:27:37.253235 24520 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.255427 29616 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.260995 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:37.281679 26344 raft_consensus.cc:1203] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:37.328796 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:37.333264 24649 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.334055 26476 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.337234 29616 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240502 21:27:37.338116 29618 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.345953 24370 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:37.359215 29885 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:37.370971 29885 tablet_bootstrap.cc:492] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Bootstrap complete.
I20240502 21:27:37.371353 29885 ts_tablet_manager.cc:1367] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.507s user 0.058s sys 0.019s
I20240502 21:27:37.372992 29860 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:37.374536 29885 raft_consensus.cc:348] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:37.377079 29885 raft_consensus.cc:729] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: LEARNER
I20240502 21:27:37.377352 29885 consensus_queue.cc:260] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:37.380002 29885 ts_tablet_manager.cc:1398] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:27:37.380957 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 427345e9a65d417c84a773ac1925a18a-9963dcea478749e0a6e076fac6537fc4 received from {username='slave'} at 127.16.122.3:58123
I20240502 21:27:37.381088 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 427345e9a65d417c84a773ac1925a18a-9963dcea478749e0a6e076fac6537fc4 on tablet 9963dcea478749e0a6e076fac6537fc4 with peer 427345e9a65d417c84a773ac1925a18a
W20240502 21:27:37.414517 24717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 78035.1us
I20240502 21:27:37.415503 29616 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:37.421028 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:37.422024 24649 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:37.429543 24242 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:37.454509 29962 tablet_copy_client.cc:308] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.5:42587
I20240502 21:27:37.454788 24520 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:37.468248 29965 tablet_copy_client.cc:308] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:37.470155 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 566929e603d54be4bf931413f0fa1d7d from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:34137)
I20240502 21:27:37.470685 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 566929e603d54be4bf931413f0fa1d7d from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:34137: session id = 7f8bb40905da4891b461220b95c66832-566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:37.475039 24649 raft_consensus.cc:1203] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240502 21:27:37.476766 24673 tablet_copy_source_session.cc:205] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:37.483225 26365 tablet_copy_service.cc:140] P b77f8f52372241d483f457389fda0f93: Received BeginTabletCopySession request for tablet bdf23b7a397441b5947ce6babe3e8f62 from peer 1398472b92c842f285b8c7cc43cc1527 ({username='slave'} at 127.16.122.4:36639)
I20240502 21:27:37.483448 26365 tablet_copy_service.cc:161] P b77f8f52372241d483f457389fda0f93: Beginning new tablet copy session on tablet bdf23b7a397441b5947ce6babe3e8f62 from peer 1398472b92c842f285b8c7cc43cc1527 at {username='slave'} at 127.16.122.4:36639: session id = 1398472b92c842f285b8c7cc43cc1527-bdf23b7a397441b5947ce6babe3e8f62
I20240502 21:27:37.487759 26365 tablet_copy_source_session.cc:205] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:37.494313 26592 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:37.494272 29966 tablet_copy_client.cc:308] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:37.497709 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 427345e9a65d417c84a773ac1925a18a ({username='slave'} at 127.16.122.3:36233)
I20240502 21:27:37.497895 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 427345e9a65d417c84a773ac1925a18a at {username='slave'} at 127.16.122.3:36233: session id = 427345e9a65d417c84a773ac1925a18a-9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:37.500947 24408 tablet_copy_source_session.cc:205] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:37.505734 29966 ts_tablet_manager.cc:1875] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:37.507422 29965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566929e603d54be4bf931413f0fa1d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.514005 24500 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:37.524322 29962 ts_tablet_manager.cc:1875] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:37.526924 26476 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:37.534641 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e"
dest_uuid: "7f8bb40905da4891b461220b95c66832"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46188
I20240502 21:27:37.534938 24520 raft_consensus.cc:595] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:37.541580 29962 ts_tablet_manager.cc:1888] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 21:27:37.546339 29962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.558171 29966 ts_tablet_manager.cc:1888] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 21:27:37.558611 29966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.561314 29962 tablet_copy_client.cc:783] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:37.562228 29965 tablet_copy_client.cc:783] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:37.562619 29965 tablet_copy_client.cc:647] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:37.565450 24632 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:37.565891 29962 tablet_copy_client.cc:647] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:37.567504 29965 tablet_copy_client.cc:523] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:37.574856 29966 tablet_copy_client.cc:783] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:37.575137 29966 tablet_copy_client.cc:647] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:37.578194 29966 tablet_copy_client.cc:523] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:37.579316 29965 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:37.592075 29962 tablet_copy_client.cc:523] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:37.593305 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:37.593302 24370 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:37.595952 29962 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:37.596138 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:37.616858 26324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:37.636715 29966 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:37.642776 29860 tablet_copy_client.cc:308] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:37.643509 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:43295)
I20240502 21:27:37.643721 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:43295: session id = 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:37.646818 24408 tablet_copy_source_session.cc:205] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:37.648352 29779 raft_consensus.cc:979] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: : Instructing follower 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:37.648531 29779 raft_consensus.cc:1067] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Signalling peer 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:37.649634 26612 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
from {username='slave'} at 127.16.122.2:56025
I20240502 21:27:37.649931 26612 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.650125 26612 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.653157 26612 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:37.658939 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:37.659165 24520 raft_consensus.cc:3032] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:37.659272 24520 raft_consensus.cc:729] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:37.659729 24520 consensus_queue.cc:260] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:37.660360 24520 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.662829 26612 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:27:37.666180 24520 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:37.667307 24782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:37.667558 24782 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.670421 24782 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:37.670790 29860 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:37.684543 29848 raft_consensus.cc:1050] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:37.685465 29848 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:37.703164 29779 raft_consensus.cc:1050] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:37.704073 29779 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:37.705555 29962 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:37.705999 29962 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap complete.
I20240502 21:27:37.706287 29962 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.110s user 0.052s sys 0.004s
I20240502 21:27:37.707202 29962 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:37.707501 29962 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: LEARNER
I20240502 21:27:37.707720 29962 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:37.708350 29962 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:37.709116 26365 tablet_copy_service.cc:342] P b77f8f52372241d483f457389fda0f93: Request end of tablet copy session 1398472b92c842f285b8c7cc43cc1527-bdf23b7a397441b5947ce6babe3e8f62 received from {username='slave'} at 127.16.122.4:36639
I20240502 21:27:37.709314 26365 tablet_copy_service.cc:433] P b77f8f52372241d483f457389fda0f93: ending tablet copy session 1398472b92c842f285b8c7cc43cc1527-bdf23b7a397441b5947ce6babe3e8f62 on tablet bdf23b7a397441b5947ce6babe3e8f62 with peer 1398472b92c842f285b8c7cc43cc1527
I20240502 21:27:37.723505 26542 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832, 9b0952b24bd8400c9940dbd5e28417b4; no voters:
I20240502 21:27:37.729056 26344 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.729669 29479 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.730060 30036 raft_consensus.cc:2781] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:37.737015 26612 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.737790 29479 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.762766 24386 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.763676 29779 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.773412 26345 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.774245 29756 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.787977 29804 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:37.795258 24520 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.797904 26612 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:37.803150 26342 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:37.807705 24242 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 7 to 8, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:37.812201 29756 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.819641 24519 consensus_queue.cc:237] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:37.822144 26344 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:37.823134 29779 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:37.833314 29804 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:37.836758 29756 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:37.839772 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f912eb05c574368909e0b3a8edf058d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:37.840603 26344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ba1af1d3624d6387bf892873f3f67e" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b77f8f52372241d483f457389fda0f93"
I20240502 21:27:37.840919 26344 raft_consensus.cc:3037] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.843950 26344 raft_consensus.cc:2454] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:37.845503 24242 catalog_manager.cc:5484] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:37.848140 24240 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:37.849830 24519 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:37.849833 26344 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:37.853026 29860 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 21:27:37.853338 29860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.854323 26741 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:37.856240 29756 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:37.860443 26741 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:37.863385 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f912eb05c574368909e0b3a8edf058d with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:37.870326 29860 tablet_copy_client.cc:783] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:37.870905 29860 tablet_copy_client.cc:647] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:37.876708 26345 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:37.879159 29860 tablet_copy_client.cc:523] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:37.893643 24649 raft_consensus.cc:1203] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:37.888468 30050 tablet_replica.cc:330] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:37.900363 30036 raft_consensus.cc:686] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Running, Role: LEADER
I20240502 21:27:37.900882 30050 raft_consensus.cc:2229] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:37.901187 30050 pending_rounds.cc:62] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Trying to abort 1 pending ops.
I20240502 21:27:37.901034 30036 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:37.901358 30050 pending_rounds.cc:68] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023350815553540096 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8f912eb05c574368909e0b3a8edf058d" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } } }
I20240502 21:27:37.902119 30050 raft_consensus.cc:2894] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 21:27:37.902302 30050 raft_consensus.cc:2866] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:37.906663 30050 raft_consensus.cc:2258] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:37.913344 24519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837"
dest_uuid: "7f8bb40905da4891b461220b95c66832"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46218
I20240502 21:27:37.913627 30050 ts_tablet_manager.cc:1875] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:37.913722 24519 raft_consensus.cc:595] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:37.915436 29860 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:27:37.929252 29479 raft_consensus.cc:979] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: : Instructing follower 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:37.929426 29479 raft_consensus.cc:1067] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Signalling peer 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:37.930456 26609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
from {username='slave'} at 127.16.122.2:56025
I20240502 21:27:37.930673 26609 raft_consensus.cc:482] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.930820 26609 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.931921 26610 raft_consensus.cc:1261] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:37.933943 26609 raft_consensus.cc:504] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:37.936823 26609 leader_election.cc:288] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:27:37.937525 29804 consensus_queue.cc:1035] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:37.939249 26609 raft_consensus.cc:2932] T 8f912eb05c574368909e0b3a8edf058d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:37.942313 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: true dest_uuid: "b77f8f52372241d483f457389fda0f93"
I20240502 21:27:37.942668 26345 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.943464 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: true dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:37.944375 24519 raft_consensus.cc:3032] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:37.945468 26345 raft_consensus.cc:2454] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:37.945936 24519 raft_consensus.cc:729] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:37.946422 24519 consensus_queue.cc:260] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:37.947095 24519 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.972282 30006 raft_consensus.cc:1050] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: attempting to promote NON_VOTER 427345e9a65d417c84a773ac1925a18a to VOTER
I20240502 21:27:37.973534 30006 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:37.980710 26612 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:37.982081 30006 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240502 21:27:37.987259 24649 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:37.988209 29621 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:37.991005 24519 raft_consensus.cc:2454] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:37.994786 26741 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:38.000540 26542 leader_election.cc:302] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7f8bb40905da4891b461220b95c66832, 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93; no voters:
I20240502 21:27:38.000892 30006 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:38.001125 30053 raft_consensus.cc:2781] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.001300 30053 raft_consensus.cc:686] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Running, Role: LEADER
I20240502 21:27:38.001871 30053 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:38.006390 29621 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.011258 30050 ts_tablet_manager.cc:1888] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:38.013093 24649 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.015547 30050 log.cc:1196] T 8f912eb05c574368909e0b3a8edf058d P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f912eb05c574368909e0b3a8edf058d
I20240502 21:27:38.016691 24242 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 12 to 13, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.020861 26609 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.026252 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 8f912eb05c574368909e0b3a8edf058d (table rebalance_test_table_3 [id=c5ed106d16b346769829dc64c1ac81d3]) successfully deleted
I20240502 21:27:38.047372 24778 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:38.052513 24649 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:38.053447 29996 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:38.058033 29996 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.058502 24649 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.063037 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a35079a106c4cca94cf65ffa93aefbf with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:38.066231 24247 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 13 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.099566 26741 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.104646 26739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc717a1b12e4380a4107e6deae2b837" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: true dest_uuid: "1769ee000eb24298a367a0de1209e6e0"
I20240502 21:27:38.104913 26739 raft_consensus.cc:3037] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.108661 26739 raft_consensus.cc:2454] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:38.113513 26740 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:38.113368 24242 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: term changed from 1 to 2, leader changed from 7f8bb40905da4891b461220b95c66832 (127.16.122.2) to 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7). New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.114567 29621 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:38.116287 26740 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.132565 24247 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: term changed from 1 to 2, leader changed from 7f8bb40905da4891b461220b95c66832 (127.16.122.2) to 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7). New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.136886 29966 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:38.137446 29966 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Bootstrap complete.
I20240502 21:27:38.138682 29966 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.502s user 0.058s sys 0.018s
I20240502 21:27:38.139578 29966 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:38.139849 29966 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: LEARNER
I20240502 21:27:38.140093 29966 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:38.141727 29966 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:38.141814 24778 raft_consensus.cc:1203] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 LEARNER]: Deduplicated request from leader. Original: 1.11->[2.12-2.12] Dedup: 2.12->[]
I20240502 21:27:38.144948 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 427345e9a65d417c84a773ac1925a18a-9d8a25580bc74004b0e31c34f841d2cc received from {username='slave'} at 127.16.122.3:36233
I20240502 21:27:38.145082 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 427345e9a65d417c84a773ac1925a18a-9d8a25580bc74004b0e31c34f841d2cc on tablet 9d8a25580bc74004b0e31c34f841d2cc with peer 427345e9a65d417c84a773ac1925a18a
I20240502 21:27:38.179438 26592 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a35079a106c4cca94cf65ffa93aefbf with delete_type TABLET_DATA_TOMBSTONED (TS 9b0952b24bd8400c9940dbd5e28417b4 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:58910
I20240502 21:27:38.235343 30079 tablet_replica.cc:330] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: stopping tablet replica
I20240502 21:27:38.235942 30079 raft_consensus.cc:2229] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.236361 30079 raft_consensus.cc:2258] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.247196 30079 ts_tablet_manager.cc:1875] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.275235 24649 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:38.278853 26741 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:38.279807 26345 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:38.282441 29848 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:38.283073 29848 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:38.284021 29860 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:38.284581 29860 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:27:38.284922 29860 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.370s user 0.055s sys 0.011s
I20240502 21:27:38.285920 29860 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:38.286202 29860 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:27:38.286108 29754 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:38.286494 29860 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:27:38.287155 29860 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240502 21:27:38.291024 30079 ts_tablet_manager.cc:1888] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240502 21:27:38.290971 26741 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:38.291564 30079 log.cc:1196] T 7a35079a106c4cca94cf65ffa93aefbf P 9b0952b24bd8400c9940dbd5e28417b4: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:27:38.292999 24233 catalog_manager.cc:4830] TS 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): tablet 7a35079a106c4cca94cf65ffa93aefbf (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:38.294026 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 received from {username='slave'} at 127.16.122.8:43295
I20240502 21:27:38.294212 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 on tablet 477d9df7d3a74b8584f3669be3256e00 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:38.297397 24240 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.297408 26345 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:38.301371 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82fb425a5430426f8030ba5497736b11 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:38.302934 24519 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:38.303364 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fb425a5430426f8030ba5497736b11 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:38.304116 30036 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:38.305300 24649 raft_consensus.cc:1203] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:38.323470 30096 tablet_replica.cc:330] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:38.325896 30096 raft_consensus.cc:2229] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.326264 30096 raft_consensus.cc:2258] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.328150 30096 ts_tablet_manager.cc:1875] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.354851 24778 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:38.355553 30036 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:38.368988 30096 ts_tablet_manager.cc:1888] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:38.369174 30096 log.cc:1196] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/82fb425a5430426f8030ba5497736b11
I20240502 21:27:38.375042 26345 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:38.376581 30036 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:38.378715 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 82fb425a5430426f8030ba5497736b11 (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
I20240502 21:27:38.414256 29965 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:38.414865 29965 tablet_bootstrap.cc:492] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:27:38.417402 29965 ts_tablet_manager.cc:1367] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.838s user 0.070s sys 0.021s
I20240502 21:27:38.418313 29965 raft_consensus.cc:348] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:38.418668 29965 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:27:38.418931 29965 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:38.419677 29965 ts_tablet_manager.cc:1398] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:38.420938 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-566929e603d54be4bf931413f0fa1d7d received from {username='slave'} at 127.16.122.2:34137
I20240502 21:27:38.421106 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 7f8bb40905da4891b461220b95c66832-566929e603d54be4bf931413f0fa1d7d on tablet 566929e603d54be4bf931413f0fa1d7d with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:27:38.429787 26609 consensus_queue.cc:237] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:38.442377 24782 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:38.443151 30036 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:38.445894 24519 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240502 21:27:38.447886 30036 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.453611 24782 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.461181 26741 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240502 21:27:38.462822 30053 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:38.463061 24240 catalog_manager.cc:5484] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 8 to 10, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.465553 26345 raft_consensus.cc:1261] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:38.465797 26741 raft_consensus.cc:1203] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:38.466310 30038 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:38.469357 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0ba1af1d3624d6387bf892873f3f67e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:38.470067 30038 consensus_queue.cc:1035] T d0ba1af1d3624d6387bf892873f3f67e P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:38.472900 26345 raft_consensus.cc:2932] T d0ba1af1d3624d6387bf892873f3f67e P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.483193 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ba1af1d3624d6387bf892873f3f67e with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:38.483754 30096 tablet_replica.cc:330] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:38.484063 30096 raft_consensus.cc:2229] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.484340 30096 raft_consensus.cc:2258] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.485873 30096 ts_tablet_manager.cc:1875] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.512632 29804 raft_consensus.cc:1050] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 427345e9a65d417c84a773ac1925a18a to VOTER
I20240502 21:27:38.513778 29804 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:38.517403 30096 ts_tablet_manager.cc:1888] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:38.521939 30096 log.cc:1196] T d0ba1af1d3624d6387bf892873f3f67e P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/d0ba1af1d3624d6387bf892873f3f67e
I20240502 21:27:38.526417 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet d0ba1af1d3624d6387bf892873f3f67e (table rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]) successfully deleted
I20240502 21:27:38.526842 24519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6"
dest_uuid: "7f8bb40905da4891b461220b95c66832"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46268
I20240502 21:27:38.527182 24519 raft_consensus.cc:595] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:38.535046 26344 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240502 21:27:38.536093 30053 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:38.552019 24649 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:38.552867 29479 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:38.578590 26476 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:38.579562 29479 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:38.597093 29479 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.607986 26344 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:38.608939 29779 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:38.614307 26344 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.625321 24649 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.636312 26476 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.646641 24247 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.653374 24519 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:38.657442 24649 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:38.660164 30155 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:38.666208 26476 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:38.667138 29479 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:38.677028 29479 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.681989 26476 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.691927 26609 consensus_queue.cc:237] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 14, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:38.705904 26345 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240502 21:27:38.706723 30053 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 21:27:38.710438 30053 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 14 to 16, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.717577 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fc717a1b12e4380a4107e6deae2b837 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:38.720589 26741 raft_consensus.cc:1261] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240502 21:27:38.721830 30053 consensus_queue.cc:1035] T 0fc717a1b12e4380a4107e6deae2b837 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 21:27:38.723934 26741 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 14 to 16, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.734561 24244 catalog_manager.cc:5484] T 0fc717a1b12e4380a4107e6deae2b837 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 14 to 16, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:38.735595 26345 raft_consensus.cc:2932] T 0fc717a1b12e4380a4107e6deae2b837 P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 14 to 16, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
W20240502 21:27:38.740518 24455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.5:42587, user_credentials={real_user=slave}} blocked reactor thread for 54470.3us
I20240502 21:27:38.743533 24649 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.745126 29779 raft_consensus.cc:979] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: : Instructing follower 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:38.745309 29779 raft_consensus.cc:1067] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Signalling peer 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:38.748417 26609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
from {username='slave'} at 127.16.122.2:56025
I20240502 21:27:38.748670 26609 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.748795 26609 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.752615 24244 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 11 to 12, VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.755959 26609 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:38.758489 26324 tablet_service.cc:1514] Processing DeleteTablet for tablet 9963dcea478749e0a6e076fac6537fc4 with delete_type TABLET_DATA_TOMBSTONED (TS b77f8f52372241d483f457389fda0f93 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:38914
I20240502 21:27:38.759585 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:38.759912 24519 raft_consensus.cc:3032] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.760021 24519 raft_consensus.cc:729] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:38.760411 24519 consensus_queue.cc:260] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:38.760987 24519 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.765826 24519 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:38.767414 26609 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Requested vote from peers 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159), 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:27:38.778980 29655 raft_consensus.cc:1050] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 427345e9a65d417c84a773ac1925a18a to VOTER
I20240502 21:27:38.780143 29655 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:38.796259 24519 raft_consensus.cc:1203] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
I20240502 21:27:38.806089 24782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:38.806478 24782 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.809190 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9963dcea478749e0a6e076fac6537fc4 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:38.810271 24782 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:38.812625 26543 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832, 9b0952b24bd8400c9940dbd5e28417b4; no voters:
I20240502 21:27:38.813149 30053 raft_consensus.cc:2781] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.813326 30053 raft_consensus.cc:686] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Running, Role: LEADER
I20240502 21:27:38.813778 30053 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:38.816033 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fc717a1b12e4380a4107e6deae2b837 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:38.816457 30096 tablet_replica.cc:330] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:38.816671 30096 raft_consensus.cc:2229] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.816871 30096 raft_consensus.cc:2258] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.817955 30096 ts_tablet_manager.cc:1875] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.825202 26476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac35b080037646b4b8b34c0eab2de4e6" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "836775e329f148e5908a1afc54f06428"
I20240502 21:27:38.825443 26476 raft_consensus.cc:3037] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.825994 30096 ts_tablet_manager.cc:1888] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20240502 21:27:38.826181 30096 log.cc:1196] T 0fc717a1b12e4380a4107e6deae2b837 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/0fc717a1b12e4380a4107e6deae2b837
I20240502 21:27:38.828663 26476 raft_consensus.cc:2454] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:38.843947 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 0fc717a1b12e4380a4107e6deae2b837 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:38.862217 30166 tablet_replica.cc:330] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: stopping tablet replica
I20240502 21:27:38.865554 24649 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:38.866420 29224 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:38.869613 26476 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:38.870250 29224 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:38.870741 26609 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:38.871577 29224 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:38.897780 29224 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.903604 24651 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.903728 26476 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.911777 24244 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.924475 24386 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:38.925797 30173 raft_consensus.cc:1050] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: attempting to promote NON_VOTER 1398472b92c842f285b8c7cc43cc1527 to VOTER
I20240502 21:27:38.926784 24651 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:38.927497 30193 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:38.928344 26476 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:38.928972 29655 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:38.931638 30061 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.932664 24240 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: term changed from 1 to 2, leader changed from 7f8bb40905da4891b461220b95c66832 (127.16.122.2) to 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7). New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.940285 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:38.941488 24651 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.944008 24240 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 11 to 12, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.941197 26476 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:38.951138 26592 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8a25580bc74004b0e31c34f841d2cc with delete_type TABLET_DATA_TOMBSTONED (TS 9b0952b24bd8400c9940dbd5e28417b4 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:58910
I20240502 21:27:38.951259 30166 raft_consensus.cc:2229] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.951601 30166 raft_consensus.cc:2258] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.952646 30166 ts_tablet_manager.cc:1875] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.959617 30173 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:38.969764 30195 tablet_replica.cc:330] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: stopping tablet replica
I20240502 21:27:38.970069 30195 raft_consensus.cc:2229] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:38.970685 30195 pending_rounds.cc:62] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Trying to abort 1 pending ops.
I20240502 21:27:38.970878 30195 pending_rounds.cc:68] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Aborting op as it isn't in flight: id { term: 1 index: 11 } timestamp: 7023350819960688640 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9d8a25580bc74004b0e31c34f841d2cc" old_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } } new_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } } }
I20240502 21:27:38.972091 30195 raft_consensus.cc:2894] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Aborting config change with OpId 1.11: Aborted: Op aborted
I20240502 21:27:38.972697 30195 raft_consensus.cc:2866] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:38.973246 30195 raft_consensus.cc:2258] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:38.974797 30195 ts_tablet_manager.cc:1875] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:38.979235 24782 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 LEARNER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:27:38.981523 24386 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:27:38.982419 30173 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:38.991721 30074 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:38.998194 30195 ts_tablet_manager.cc:1888] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 21:27:39.000296 30195 log.cc:1196] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/wals/9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:39.001323 24233 catalog_manager.cc:4830] TS 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): tablet 9d8a25580bc74004b0e31c34f841d2cc (table rebalance_test_table_1 [id=57c59a3b5480478d9f9953356d0daddc]) successfully deleted
I20240502 21:27:39.007424 26476 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:27:39.008281 30173 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:39.012395 24384 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b"
dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38450
I20240502 21:27:39.012676 24384 raft_consensus.cc:595] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:39.019433 30166 ts_tablet_manager.cc:1888] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 21:27:39.019840 30166 log.cc:1196] T 9963dcea478749e0a6e076fac6537fc4 P b77f8f52372241d483f457389fda0f93: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-4/wal/wals/9963dcea478749e0a6e076fac6537fc4
I20240502 21:27:39.020900 30164 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.035183 24386 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.039749 24232 catalog_manager.cc:4830] TS b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): tablet 9963dcea478749e0a6e076fac6537fc4 (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:39.039320 24778 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.041707 24244 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 11 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b77f8f52372241d483f457389fda0f93" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.052232 26476 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.059536 26345 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:39.063135 24778 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:27:39.063957 30164 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20240502 21:27:39.067479 26476 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:27:39.068558 30164 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:39.071740 30074 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.071991 24778 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.082823 24240 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 13 to 14, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 2 leader_uuid: "b77f8f52372241d483f457389fda0f93" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.083969 30061 raft_consensus.cc:1050] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:27:39.085129 30061 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:27:39.089051 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet bdf23b7a397441b5947ce6babe3e8f62 with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:39.077992 26479 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.096531 24651 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d"
dest_uuid: "427345e9a65d417c84a773ac1925a18a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45688
I20240502 21:27:39.096932 24651 raft_consensus.cc:595] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:39.100354 26741 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:39.103612 29655 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:39.106678 24778 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:39.107740 29655 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:39.112396 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:39.113260 30061 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:39.131475 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdf23b7a397441b5947ce6babe3e8f62 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:39.142287 30061 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:39.144743 24782 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:39.152344 29655 raft_consensus.cc:979] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: : Instructing follower 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:39.152547 30193 raft_consensus.cc:1067] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Signalling peer 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:39.153549 26609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
from {username='slave'} at 127.16.122.1:36393
I20240502 21:27:39.153867 26609 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.154009 26609 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.161630 26609 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:39.163657 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:39.163905 24783 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.174551 24783 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:39.175359 29848 raft_consensus.cc:979] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: : Instructing follower 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:39.175503 29848 raft_consensus.cc:1067] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Signalling peer 9b0952b24bd8400c9940dbd5e28417b4 to start an election
I20240502 21:27:39.176532 26610 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
from {username='slave'} at 127.16.122.3:45727
I20240502 21:27:39.176785 26610 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.176898 26610 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.180950 26610 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:39.182240 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:39.182466 24651 raft_consensus.cc:3032] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:39.182574 24651 raft_consensus.cc:729] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:39.182957 24651 consensus_queue.cc:260] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:39.183616 24651 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.189136 26610 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045)
I20240502 21:27:39.190011 26741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566929e603d54be4bf931413f0fa1d7d" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "1769ee000eb24298a367a0de1209e6e0"
I20240502 21:27:39.190228 26741 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.194054 26741 raft_consensus.cc:2454] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:39.196861 26542 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4; no voters:
I20240502 21:27:39.197384 30036 raft_consensus.cc:2781] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:39.197551 30036 raft_consensus.cc:686] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Running, Role: LEADER
I20240502 21:27:39.197999 30036 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:39.214624 24651 raft_consensus.cc:2454] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:39.229176 24244 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: term changed from 1 to 2, leader changed from 427345e9a65d417c84a773ac1925a18a (127.16.122.3) to 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7). New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.237850 24386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:39.238103 24386 raft_consensus.cc:3032] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:39.238227 24386 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Running, Role: LEADER
I20240502 21:27:39.238286 26345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "9b0952b24bd8400c9940dbd5e28417b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b77f8f52372241d483f457389fda0f93"
I20240502 21:27:39.238631 26345 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.238585 24386 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:39.239104 24386 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.236637 24240 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 10 to 11, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:39.242527 26345 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:39.243186 26542 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93; no voters:
I20240502 21:27:39.243676 30036 raft_consensus.cc:2781] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:39.243695 24386 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0952b24bd8400c9940dbd5e28417b4 in term 2.
I20240502 21:27:39.243849 30036 raft_consensus.cc:686] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Running, Role: LEADER
I20240502 21:27:39.244230 30036 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:39.246498 26609 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [CANDIDATE]: Term 2 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:27:39.250756 26741 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:39.254493 24242 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: term changed from 1 to 2, leader changed from e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) to 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7). New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.260028 26612 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:27:39.278129 30216 tablet_replica.cc:330] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:39.279989 30216 raft_consensus.cc:2229] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:39.282049 30216 raft_consensus.cc:2258] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:39.287640 30216 ts_tablet_manager.cc:1875] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:39.304266 26479 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:27:39.305714 30036 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:39.312796 24519 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:27:39.313643 30036 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:39.315222 24764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:39.373678 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.432509 24778 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 21:27:39.433339 30036 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:39.455714 26458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.478331 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:39.482313 30216 ts_tablet_manager.cc:1888] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 21:27:39.482839 30216 log.cc:1196] T bdf23b7a397441b5947ce6babe3e8f62 P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdf23b7a397441b5947ce6babe3e8f62
I20240502 21:27:39.483866 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet bdf23b7a397441b5947ce6babe3e8f62 (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
I20240502 21:27:39.494184 26612 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:39.500531 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:39.517123 24632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:39.536309 24783 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240502 21:27:39.537396 30036 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:39.541162 30053 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.549188 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:39.550653 24240 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 13 to 15, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.552393 24778 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.556604 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet ac35b080037646b4b8b34c0eab2de4e6 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:39.569880 30299 tablet_replica.cc:330] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:39.570294 30299 raft_consensus.cc:2229] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:39.570612 30299 raft_consensus.cc:2258] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:39.572391 30299 ts_tablet_manager.cc:1875] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:39.581290 30299 ts_tablet_manager.cc:1888] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.14
I20240502 21:27:39.586295 30299 log.cc:1196] T ac35b080037646b4b8b34c0eab2de4e6 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac35b080037646b4b8b34c0eab2de4e6
I20240502 21:27:39.587471 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet ac35b080037646b4b8b34c0eab2de4e6 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:39.593502 24500 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:39.627772 24651 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240502 21:27:39.630187 30053 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 21:27:39.645377 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240502 21:27:39.652192 30038 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
I20240502 21:27:39.654492 26345 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:39.655352 30053 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:39.713855 24386 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:39.714962 30053 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:39.722533 24519 raft_consensus.cc:3037] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Advancing to term 2
I20240502 21:27:39.733604 24519 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 LEARNER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240502 21:27:39.734874 30053 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 21:27:39.738973 30038 raft_consensus.cc:1050] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4: attempting to promote NON_VOTER 7f8bb40905da4891b461220b95c66832 to VOTER
I20240502 21:27:39.739874 30038 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:39.744285 24519 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 LEARNER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240502 21:27:39.746086 30038 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240502 21:27:39.763926 24783 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:39.764801 30325 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:39.771533 26740 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240502 21:27:39.772322 30325 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 21:27:39.775283 24782 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0"
dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43174
I20240502 21:27:39.775607 24782 raft_consensus.cc:595] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:39.795547 26740 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 16 to 18, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:39.799113 26324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:39.791661 30325 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.18: config changed from index 16 to 18, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:39.813449 24242 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 16 to 18, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.821592 30158 raft_consensus.cc:979] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: : Instructing follower 836775e329f148e5908a1afc54f06428 to start an election
I20240502 21:27:39.821769 30158 raft_consensus.cc:1067] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Signalling peer 836775e329f148e5908a1afc54f06428 to start an election
I20240502 21:27:39.823845 26477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0"
dest_uuid: "836775e329f148e5908a1afc54f06428"
from {username='slave'} at 127.16.122.4:34569
I20240502 21:27:39.824149 26477 raft_consensus.cc:482] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.824294 26477 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.828960 26477 raft_consensus.cc:504] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:39.831678 24519 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 16 to 18, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:39.842681 26477 leader_election.cc:288] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [CANDIDATE]: Term 2 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:39.842813 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "836775e329f148e5908a1afc54f06428" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7f8bb40905da4891b461220b95c66832"
I20240502 21:27:39.843043 24520 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.843757 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "836775e329f148e5908a1afc54f06428" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:39.844034 24783 raft_consensus.cc:3032] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:39.844125 24783 raft_consensus.cc:729] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:27:39.844462 24783 consensus_queue.cc:260] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:39.844951 24783 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.845890 24520 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836775e329f148e5908a1afc54f06428 in term 2.
I20240502 21:27:39.855459 24783 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836775e329f148e5908a1afc54f06428 in term 2.
I20240502 21:27:39.855999 26412 leader_election.cc:302] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, 7f8bb40905da4891b461220b95c66832, 836775e329f148e5908a1afc54f06428; no voters:
I20240502 21:27:39.864449 30340 raft_consensus.cc:2781] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:39.864826 30340 raft_consensus.cc:686] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 2 LEADER]: Becoming Leader. State: Replica: 836775e329f148e5908a1afc54f06428, State: Running, Role: LEADER
I20240502 21:27:39.863869 24651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "836775e329f148e5908a1afc54f06428" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:39.866040 24651 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:39.865852 26612 consensus_queue.cc:237] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:39.866099 30340 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:39.869069 24651 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836775e329f148e5908a1afc54f06428 in term 2.
I20240502 21:27:39.881367 26741 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20240502 21:27:39.882212 30325 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240502 21:27:39.899680 24244 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 reported cstate change: term changed from 1 to 2, leader changed from 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) to 836775e329f148e5908a1afc54f06428 (127.16.122.6). New cstate: current_term: 2 leader_uuid: "836775e329f148e5908a1afc54f06428" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.912103 24648 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240502 21:27:39.913803 26476 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240502 21:27:39.916164 30325 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:27:39.918509 26476 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 836775e329f148e5908a1afc54f06428 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:39.930339 30036 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.19: config changed from index 18 to 19, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:39.939046 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 566929e603d54be4bf931413f0fa1d7d with cas_config_opid_index 18: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:39.941293 26740 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:39.942189 24520 raft_consensus.cc:1261] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20240502 21:27:39.944691 24244 catalog_manager.cc:5484] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 18 to 19, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.948158 30325 consensus_queue.cc:1035] T 566929e603d54be4bf931413f0fa1d7d P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.003s
I20240502 21:27:39.954430 24519 raft_consensus.cc:2932] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:39.979372 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet 566929e603d54be4bf931413f0fa1d7d with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 19) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:40.017948 30364 tablet_replica.cc:330] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:40.023453 30364 raft_consensus.cc:2229] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:40.024122 30364 raft_consensus.cc:2258] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:40.027972 30364 ts_tablet_manager.cc:1875] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:40.042917 30364 ts_tablet_manager.cc:1888] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.17
I20240502 21:27:40.043095 30364 log.cc:1196] T 566929e603d54be4bf931413f0fa1d7d P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/566929e603d54be4bf931413f0fa1d7d
I20240502 21:27:40.047950 24632 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:40.056152 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet 566929e603d54be4bf931413f0fa1d7d (table rebalance_test_table_8 [id=60beec8dbb7f48e1be85a9508b5b7fcd]) successfully deleted
I20240502 21:27:40.072494 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.090584 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:40.169721 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:40.180758 24500 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:40.261088 26612 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:40.263651 26345 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:40.264528 30325 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:40.268321 30053 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:40.271302 26345 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:40.281786 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b6a4a74e0544e0e99e734a145c91d3b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:40.289528 24244 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 8 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:40.295586 24370 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6a4a74e0544e0e99e734a145c91d3b with delete_type TABLET_DATA_TOMBSTONED (TS e856847c3f494e6aa57f9cecf117ce4d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35166
I20240502 21:27:40.329468 30384 tablet_replica.cc:330] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: stopping tablet replica
I20240502 21:27:40.330087 30384 raft_consensus.cc:2229] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:40.330613 30384 raft_consensus.cc:2258] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:40.332748 30384 ts_tablet_manager.cc:1875] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:40.361500 24778 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:40.362332 30053 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:40.364600 24778 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:40.370139 26458 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:40.379575 26324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:40.407094 30384 ts_tablet_manager.cc:1888] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:40.407271 30384 log.cc:1196] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-0/wal/wals/2b6a4a74e0544e0e99e734a145c91d3b
I20240502 21:27:40.408128 24233 catalog_manager.cc:4830] TS e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): tablet 2b6a4a74e0544e0e99e734a145c91d3b (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
I20240502 21:27:40.413813 24778 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 836775e329f148e5908a1afc54f06428: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:40.415493 30340 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:40.417851 24519 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Refusing update from remote peer 836775e329f148e5908a1afc54f06428: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:40.422931 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:40.433018 30392 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:40.467753 24649 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Refusing update from remote peer 836775e329f148e5908a1afc54f06428: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:40.468649 30340 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 21:27:40.475665 24455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.8:37045, user_credentials={real_user=slave}} blocked reactor thread for 100714us
I20240502 21:27:40.527122 26479 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:40.531071 24519 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Refusing update from remote peer 836775e329f148e5908a1afc54f06428: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:40.531641 30392 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:40.534853 30392 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:40.539302 24520 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:40.546643 24240 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 8 to 10, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New cstate: current_term: 2 leader_uuid: "836775e329f148e5908a1afc54f06428" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:40.559671 24648 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Refusing update from remote peer 836775e329f148e5908a1afc54f06428: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:40.562278 30340 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 836775e329f148e5908a1afc54f06428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:40.564648 24649 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:40.566504 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:40.568203 24764 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with delete_type TABLET_DATA_TOMBSTONED (TS 1398472b92c842f285b8c7cc43cc1527 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34286
I20240502 21:27:40.618867 30420 tablet_replica.cc:330] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: stopping tablet replica
I20240502 21:27:40.619741 30420 raft_consensus.cc:2229] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:40.623158 30420 pending_rounds.cc:62] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Trying to abort 1 pending ops.
I20240502 21:27:40.624120 30420 pending_rounds.cc:68] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023350824464588800 op_type: NO_OP noop_request { }
I20240502 21:27:40.624372 30420 raft_consensus.cc:2866] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 21:27:40.625144 30420 raft_consensus.cc:2258] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:40.626592 30420 ts_tablet_manager.cc:1875] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:40.644009 30420 ts_tablet_manager.cc:1888] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:40.645675 30420 log.cc:1196] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 1398472b92c842f285b8c7cc43cc1527: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-3/wal/wals/a8a9b14a7ff04ddab0d1e1a8e9fcb9b0
I20240502 21:27:40.646800 24233 catalog_manager.cc:4830] TS 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 (table rebalance_test_table_5 [id=915a69d8d0034f63a71fc65a9234e61e]) successfully deleted
I20240502 21:27:40.659708 24386 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
W20240502 21:27:40.662961 24323 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bf18ffb96d04d5e9ec8ee9563812ba0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:40.664896 24649 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:40.665696 29224 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:40.669669 26741 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:40.670774 30388 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:40.673866 30388 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:40.678340 24649 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:40.685523 26741 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:40.692555 24240 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:40.785145 24649 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:40.792995 26741 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 21:27:40.793298 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:40.794062 30336 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:40.833629 30336 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:40.839327 26740 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:40.847218 24240 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:40.862725 26345 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:40.871714 30481 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:40.873610 24386 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:40.886533 26345 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
W20240502 21:27:40.891556 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:40.896642 24649 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:40.897475 30438 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:40.907117 26479 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:40.909087 30388 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:40.910231 30388 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:40.919193 24649 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:40.925868 26479 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:40.933048 24240 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:27:41.006824 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } }
I20240502 21:27:41.015487 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:41.015493 26741 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:41.016237 30438 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:41.016620 29224 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:41.017746 24778 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:41.019574 29224 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:41.063835 30438 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } }
I20240502 21:27:41.074906 26741 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } }
I20240502 21:27:41.080760 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } }
I20240502 21:27:41.108357 24244 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 11 to 12. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } }
I20240502 21:27:41.131930 26612 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } }
I20240502 21:27:41.148029 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:41.150171 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:41.150890 30438 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:41.153477 24778 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:41.154682 30388 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:41.155282 30388 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:41.159057 24778 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:41.169570 26612 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:41.181694 24240 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:41.186933 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 477d9df7d3a74b8584f3669be3256e00 with delete_type TABLET_DATA_TOMBSTONED (TS 1769ee000eb24298a367a0de1209e6e0 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:41.187191 30577 ts_tablet_manager.cc:902] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:41.195155 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:41.205205 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
W20240502 21:27:41.212831 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477d9df7d3a74b8584f3669be3256e00. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:41.213588 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:41.214258 30577 tablet_copy_client.cc:308] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:41.214572 30438 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:41.215042 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 836775e329f148e5908a1afc54f06428 ({username='slave'} at 127.16.122.6:45315)
I20240502 21:27:41.215204 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 836775e329f148e5908a1afc54f06428 at {username='slave'} at 127.16.122.6:45315: session id = 836775e329f148e5908a1afc54f06428-7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:41.216401 24778 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:41.217353 30388 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:41.220579 30438 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:41.223856 24408 tablet_copy_source_session.cc:205] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:41.226848 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:27:41.233269 24240 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 13 to 14, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.233884 30577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:41.255075 26612 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:41.257975 24778 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } }
I20240502 21:27:41.266023 30578 tablet_replica.cc:330] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:41.267685 30578 raft_consensus.cc:2229] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:41.269413 30578 raft_consensus.cc:2258] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:41.276800 30577 tablet_copy_client.cc:783] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:41.277148 30577 tablet_copy_client.cc:647] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:41.279402 30577 tablet_copy_client.cc:523] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:41.284709 30577 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: Bootstrap starting.
I20240502 21:27:41.285171 30578 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:41.306807 24386 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:41.309286 26741 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 21:27:41.309859 24324 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): Couldn't send request to peer 427345e9a65d417c84a773ac1925a18a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:41.309942 30438 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:41.310688 26612 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:41.312283 30388 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:41.324748 30602 ts_tablet_manager.cc:902] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:41.347949 30438 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:41.351526 26741 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:41.355990 30578 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:27:41.356658 30578 log.cc:1196] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/477d9df7d3a74b8584f3669be3256e00
I20240502 21:27:41.357864 24232 catalog_manager.cc:4830] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): tablet 477d9df7d3a74b8584f3669be3256e00 (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
I20240502 21:27:41.371728 24240 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.388873 26609 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:27:41.413836 24649 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:41.431751 26609 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 21:27:41.433527 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049bec57d2474c328d2e26e16a4d33a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:41.433861 30480 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:41.435650 30602 tablet_copy_client.cc:308] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:41.437368 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 82fb425a5430426f8030ba5497736b11 from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:34137)
I20240502 21:27:41.437534 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 82fb425a5430426f8030ba5497736b11 from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:34137: session id = 7f8bb40905da4891b461220b95c66832-82fb425a5430426f8030ba5497736b11
I20240502 21:27:41.437786 26479 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:41.439932 30480 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:41.440698 24673 tablet_copy_source_session.cc:205] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:41.447474 30602 ts_tablet_manager.cc:1875] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:41.452036 30630 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:41.452813 26479 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:41.459564 24240 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 9 to 10, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:41.474427 30602 ts_tablet_manager.cc:1888] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 21:27:41.484558 30602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb425a5430426f8030ba5497736b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:41.492319 26609 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:27:41.495894 30629 ts_tablet_manager.cc:902] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:41.506008 30602 tablet_copy_client.cc:783] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:41.506546 30602 tablet_copy_client.cc:647] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:41.509157 30602 tablet_copy_client.cc:523] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:41.517738 30602 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:27:41.522490 24386 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
W20240502 21:27:41.528888 24322 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bf18ffb96d04d5e9ec8ee9563812ba0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:41.531066 26741 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:27:41.531869 24323 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:41.532297 30388 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:41.535751 30388 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:41.539109 26741 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:41.544292 24649 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:41.544989 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:41.547149 24649 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:41.555124 30577 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:41.555663 30577 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: Bootstrap complete.
I20240502 21:27:41.555992 30577 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: Time spent bootstrapping tablet: real 0.271s user 0.061s sys 0.005s
I20240502 21:27:41.563668 24240 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 7 to 8, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.567121 30577 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:41.567417 30577 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 836775e329f148e5908a1afc54f06428, State: Initialized, Role: LEARNER
I20240502 21:27:41.567642 30577 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } }
I20240502 21:27:41.569334 30577 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:41.574054 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 836775e329f148e5908a1afc54f06428-7bf18ffb96d04d5e9ec8ee9563812ba0 received from {username='slave'} at 127.16.122.6:45315
I20240502 21:27:41.574234 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 836775e329f148e5908a1afc54f06428-7bf18ffb96d04d5e9ec8ee9563812ba0 on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with peer 836775e329f148e5908a1afc54f06428
I20240502 21:27:41.609892 30629 tablet_copy_client.cc:308] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:41.610718 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 9b0952b24bd8400c9940dbd5e28417b4 ({username='slave'} at 127.16.122.7:38507)
I20240502 21:27:41.610917 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 9d8a25580bc74004b0e31c34f841d2cc from peer 9b0952b24bd8400c9940dbd5e28417b4 at {username='slave'} at 127.16.122.7:38507: session id = 9b0952b24bd8400c9940dbd5e28417b4-9d8a25580bc74004b0e31c34f841d2cc
I20240502 21:27:41.614217 24408 tablet_copy_source_session.cc:205] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:41.619663 30629 ts_tablet_manager.cc:1875] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:41.633071 30629 ts_tablet_manager.cc:1888] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240502 21:27:41.633653 30629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8a25580bc74004b0e31c34f841d2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:41.637414 30629 tablet_copy_client.cc:783] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:41.637764 30629 tablet_copy_client.cc:647] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:41.642062 30629 tablet_copy_client.cc:523] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:41.646553 30629 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap starting.
I20240502 21:27:41.665622 30577 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 836775e329f148e5908a1afc54f06428: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
F20240502 21:27:41.665962 30577 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.queue_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.queue_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_cpu_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_cpu_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_wall_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_wall_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us) (tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.queue_time_us, tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.queue_time_us) (tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_cpu_time_us, tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_cpu_time_us) (tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_wall_time_us, tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_wall_time_us) (tablet-download-pool-2ee3718b120e4fad9ae03e47d3f1fffa.queue_time_us, tablet-download-pool-2ee3718b120e4fad9ae03e47d3f1fffa.queue_time_us) (tablet-download-pool-2ee3718b120e4fad9ae03e47d3f1fffa.run_cpu_time_us, tablet-download-pool-2ee3718b120e4fad9ae03e47d3f1fffa.run_cpu_time_us) (tablet-download-pool-2ee3718b120e4fad9ae03e47d3f1fffa.run_wall_time_us, tablet-download-pool-2ee3718b120e4fad9ae03e47d3f1fffa.run_wall_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.queue_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.queue_time_us) (tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.queue_time_us, tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.queue_time_us) (tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.run_cpu_time_us, tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.run_cpu_time_us) (tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.run_wall_time_us, tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.run_wall_time_us) (tablet-download-pool-5c6e7eba855e48c0b0e18fa363ea01d4.queue_time_us, tablet-download-pool-5c6e7eba855e48c0b0e18fa363ea01d4.queue_time_us) (tablet-download-pool-5c6e7eba855e48c0b0e18fa363ea01d4.run_cpu_time_us, tablet-download-pool-5c6e7eba855e48c0b0e18fa363ea01d4.run_cpu_time_us) (tablet-download-pool-5c6e7eba855e48c0b0e18fa363ea01d4.run_wall_time_us, tablet-download-pool-5c6e7eba855e48c0b0e18fa363ea01d4.run_wall_time_us) (tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.queue_time_us, tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.queue_time_us) (tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_cpu_time_us, tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_cpu_time_us) (tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_wall_time_us, tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_wall_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.queue_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.queue_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_cpu_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_cpu_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_wall_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_wall_time_us) (tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.queue_time_us, tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.queue_time_us) (tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_cpu_time_us, tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_cpu_time_us) (tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_wall_time_us, tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_wall_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_cpu_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_cpu_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_wall_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_wall_time_us) (tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.queue_time_us, tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.queue_time_us) (tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_cpu_time_us, tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_cpu_time_us) (tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_wall_time_us, tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_wall_time_us) (tablet-download-pool-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0.queue_time_us, tablet-download-pool-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0.queue_time_us) (tablet-download-pool-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0.run_cpu_time_us, tablet-download-pool-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0.run_cpu_time_us) (tablet-download-pool-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0.run_wall_time_us, tablet-download-pool-a8a9b14a7ff04ddab0d1e1a8e9fcb9b0.run_wall_time_us) (tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.queue_time_us, tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.queue_time_us) (tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_cpu_time_us, tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_cpu_time_us) (tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_wall_time_us, tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_wall_time_us) (tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.queue_time_us, tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.queue_time_us) (tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_cpu_time_us, tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_cpu_time_us) (tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_wall_time_us, tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fdc6158ddcd google::LogMessage::Fail() at ??:0
@ 0x7fdc61591b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdc6158d7cc google::LogMessage::Flush() at ??:0
@ 0x7fdc6158ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdc642ce742 kudu::TraceMetrics::InternName() at ??:0
I20240502 21:27:41.750187 24649 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95"
dest_uuid: "427345e9a65d417c84a773ac1925a18a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45822
I20240502 21:27:41.750504 24649 raft_consensus.cc:595] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Received request to transfer leadership
@ 0x7fdc6429af8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 21:27:41.791711 30480 raft_consensus.cc:979] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: : Instructing follower 7f8bb40905da4891b461220b95c66832 to start an election
I20240502 21:27:41.791896 30480 raft_consensus.cc:1067] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Signalling peer 7f8bb40905da4891b461220b95c66832 to start an election
I20240502 21:27:41.792639 24519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95"
dest_uuid: "7f8bb40905da4891b461220b95c66832"
from {username='slave'} at 127.16.122.3:58123
I20240502 21:27:41.792886 24519 raft_consensus.cc:482] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.792987 24519 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7fdc64294cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 21:27:41.797132 24519 raft_consensus.cc:504] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:41.798589 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:41.798856 24649 raft_consensus.cc:3032] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:41.798951 24649 raft_consensus.cc:729] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Running, Role: LEADER
I20240502 21:27:41.799289 24649 consensus_queue.cc:260] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:41.799793 24649 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:41.803421 24649 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 2.
I20240502 21:27:41.804767 26741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1769ee000eb24298a367a0de1209e6e0"
I20240502 21:27:41.805024 26741 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:41.808257 26741 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 2.
@ 0x7fdc736e78de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 21:27:41.808805 24455 leader_election.cc:302] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1769ee000eb24298a367a0de1209e6e0, 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters:
I20240502 21:27:41.809114 30544 raft_consensus.cc:2781] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:41.809240 30544 raft_consensus.cc:686] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 2 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:41.809507 30544 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:41.778280 30602 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:41.811983 30602 tablet_bootstrap.cc:492] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:27:41.812922 30602 ts_tablet_manager.cc:1367] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.295s user 0.063s sys 0.009s
I20240502 21:27:41.813097 26609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5c40e2bf0340958ecd3e7541209e95" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
I20240502 21:27:41.813373 26609 raft_consensus.cc:3037] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:41.814249 30602 raft_consensus.cc:348] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:41.814513 30602 raft_consensus.cc:729] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:27:41.814689 30602 consensus_queue.cc:260] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:41.816485 26609 raft_consensus.cc:2454] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 2.
I20240502 21:27:41.817554 24519 leader_election.cc:288] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 2 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
I20240502 21:27:41.818300 30602 ts_tablet_manager.cc:1398] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:41.819180 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-82fb425a5430426f8030ba5497736b11 received from {username='slave'} at 127.16.122.2:34137
I20240502 21:27:41.819322 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session 7f8bb40905da4891b461220b95c66832-82fb425a5430426f8030ba5497736b11 on tablet 82fb425a5430426f8030ba5497736b11 with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:27:41.835232 24244 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 1 to 2, leader changed from 427345e9a65d417c84a773ac1925a18a (127.16.122.3) to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 2 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
@ 0x7fdc736ffdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 21:27:41.888084 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:41.909557 30727 ts_tablet_manager.cc:902] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
I20240502 21:27:41.916067 30727 tablet_copy_client.cc:308] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.3:38427
I20240502 21:27:41.918094 24673 tablet_copy_service.cc:140] P 427345e9a65d417c84a773ac1925a18a: Received BeginTabletCopySession request for tablet 049bec57d2474c328d2e26e16a4d33a9 from peer b77f8f52372241d483f457389fda0f93 ({username='slave'} at 127.16.122.5:36629)
I20240502 21:27:41.918303 24673 tablet_copy_service.cc:161] P 427345e9a65d417c84a773ac1925a18a: Beginning new tablet copy session on tablet 049bec57d2474c328d2e26e16a4d33a9 from peer b77f8f52372241d483f457389fda0f93 at {username='slave'} at 127.16.122.5:36629: session id = b77f8f52372241d483f457389fda0f93-049bec57d2474c328d2e26e16a4d33a9
I20240502 21:27:41.922641 24673 tablet_copy_source_session.cc:205] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:41.935591 30747 ts_tablet_manager.cc:902] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:41.937217 30727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049bec57d2474c328d2e26e16a4d33a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:42.027491 30727 tablet_copy_client.cc:783] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:42.027884 30727 tablet_copy_client.cc:647] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: tablet copy: Starting download of 1 WAL segments...
@ 0x7fdc7382a480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 21:27:42.046005 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:42.051124 26476 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:42.052022 30602 ts_tablet_manager.cc:902] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:42.058909 24632 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:42.065373 30727 tablet_copy_client.cc:523] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:42.104535 24632 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:42.107401 30747 tablet_copy_client.cc:308] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:42.108167 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 98b5668e0d504db5a7874f041c14dbfa from peer 427345e9a65d417c84a773ac1925a18a ({username='slave'} at 127.16.122.3:36233)
I20240502 21:27:42.108343 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 98b5668e0d504db5a7874f041c14dbfa from peer 427345e9a65d417c84a773ac1925a18a at {username='slave'} at 127.16.122.3:36233: session id = 427345e9a65d417c84a773ac1925a18a-98b5668e0d504db5a7874f041c14dbfa
I20240502 21:27:42.111328 24408 tablet_copy_source_session.cc:205] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:42.112506 30747 ts_tablet_manager.cc:1875] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:42.131062 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:42.140609 30747 ts_tablet_manager.cc:1888] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 21:27:42.141989 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:42.159583 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:42.167241 24500 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:42.193645 24500 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:42.194011 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:42.201503 30727 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: Bootstrap starting.
I20240502 21:27:42.211643 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:42.226441 30747 tablet_copy_client.cc:783] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:42.226948 30747 tablet_copy_client.cc:647] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:42.234453 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:42.237558 30747 tablet_copy_client.cc:523] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:42.251647 24649 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:42.252249 30544 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7fdc73862a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 21:27:42.262719 26458 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:42.280326 26324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:42.280901 26325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:42.294873 26458 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:42.298452 30602 tablet_copy_client.cc:308] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:42.299171 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:59747)
I20240502 21:27:42.299376 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:59747: session id = 7f8bb40905da4891b461220b95c66832-7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:42.302584 24408 tablet_copy_source_session.cc:205] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:42.315086 30602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:42.333098 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:42.343814 26741 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:42.347548 30689 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7fdc73845981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 21:27:42.400197 26609 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:42.400964 30689 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:42.426127 30602 tablet_copy_client.cc:783] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:42.426888 30602 tablet_copy_client.cc:647] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:42.428964 30602 tablet_copy_client.cc:523] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:42.433212 30602 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
W20240502 21:27:42.451939 24457 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.122.3:38427, user_credentials={real_user=slave}} blocked reactor thread for 77192.4us
I20240502 21:27:42.499372 24519 raft_consensus.cc:1203] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:42.502373 24519 consensus_queue.cc:237] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:42.507956 26609 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:42.509810 30824 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:42.513367 30689 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:42.519268 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc5c40e2bf0340958ecd3e7541209e95 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:42.521337 24244 catalog_manager.cc:5484] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 8 to 10, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New cstate: current_term: 2 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:42.523464 30747 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:27:42.552165 24632 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5c40e2bf0340958ecd3e7541209e95 with delete_type TABLET_DATA_TOMBSTONED (TS 427345e9a65d417c84a773ac1925a18a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41984
I20240502 21:27:42.578852 30840 tablet_replica.cc:330] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: stopping tablet replica
I20240502 21:27:42.579066 30840 raft_consensus.cc:2229] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:42.579262 30840 pending_rounds.cc:62] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Trying to abort 1 pending ops.
I20240502 21:27:42.579355 30840 pending_rounds.cc:68] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023350832375181312 op_type: NO_OP noop_request { }
I20240502 21:27:42.579495 30840 raft_consensus.cc:2866] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 21:27:42.579573 30840 raft_consensus.cc:2258] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:42.581341 30840 ts_tablet_manager.cc:1875] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:42.587474 30840 ts_tablet_manager.cc:1888] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:42.587579 30840 log.cc:1196] T dc5c40e2bf0340958ecd3e7541209e95 P 427345e9a65d417c84a773ac1925a18a: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc5c40e2bf0340958ecd3e7541209e95
I20240502 21:27:42.595531 26609 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:42.605144 26741 raft_consensus.cc:1261] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:42.613029 30544 consensus_queue.cc:1035] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:42.615197 26741 raft_consensus.cc:2932] T dc5c40e2bf0340958ecd3e7541209e95 P 1769ee000eb24298a367a0de1209e6e0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:42.618055 24234 catalog_manager.cc:4830] TS 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): tablet dc5c40e2bf0340958ecd3e7541209e95 (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
I20240502 21:27:42.632551 30438 raft_consensus.cc:1050] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 836775e329f148e5908a1afc54f06428 to VOTER
I20240502 21:27:42.633836 30438 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
W20240502 21:27:42.704591 24322 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:42.707000 24651 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:42.708526 26476 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:42.709370 30388 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:42.710064 30388 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7fdc7384570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 21:27:42.717581 30629 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:42.718034 30629 tablet_bootstrap.cc:492] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Bootstrap complete.
I20240502 21:27:42.719888 26741 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:42.720582 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:42.730123 30388 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
@ 0x7fdc6429eb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 21:27:42.742127 26740 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:42.751896 24244 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 reported cstate change: config changed from index 8 to 9, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:42.718330 30629 ts_tablet_manager.cc:1367] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Time spent bootstrapping tablet: real 1.072s user 0.069s sys 0.015s
@ 0x7fdc642a219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 21:27:42.763511 30629 raft_consensus.cc:348] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:42.784499 24651 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:27:42.787706 26476 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 836775e329f148e5908a1afc54f06428 (127.16.122.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
@ 0x7fdc642a2019 std::_Function_handler<>::_M_invoke() at ??:0
I20240502 21:27:42.802218 24384 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:27:42.804549 30629 raft_consensus.cc:729] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Initialized, Role: LEARNER
I20240502 21:27:42.804944 30629 consensus_queue.cc:260] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:27:42.806608 26741 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:42.807202 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 21:27:42.810496 26476 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:42.811095 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:42.814514 26475 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:42.816398 30629 ts_tablet_manager.cc:1398] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4: Time spent starting tablet: real 0.054s user 0.001s sys 0.002s
I20240502 21:27:42.820012 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-9d8a25580bc74004b0e31c34f841d2cc received from {username='slave'} at 127.16.122.7:38507
I20240502 21:27:42.820122 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 9b0952b24bd8400c9940dbd5e28417b4-9d8a25580bc74004b0e31c34f841d2cc on tablet 9d8a25580bc74004b0e31c34f841d2cc with peer 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:42.827173 26609 raft_consensus.cc:1203] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240502 21:27:42.829788 24242 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 836775e329f148e5908a1afc54f06428 reported cstate change: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:42.834064 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with delete_type TABLET_DATA_TOMBSTONED (TS 7f8bb40905da4891b461220b95c66832 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57654
@ 0x7fdc6427d01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc676026db start_thread at ??:0
I20240502 21:27:42.830991 30388 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
@ 0x7fdc6044071f clone at ??:0
I20240502 21:27:42.858084 30630 raft_consensus.cc:1050] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER 7f8bb40905da4891b461220b95c66832 to VOTER
I20240502 21:27:42.859103 30630 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:42.865170 26345 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:42.866165 24519 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:42.866830 30635 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:42.867293 30635 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:42.872476 26741 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:42.873210 30480 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:27:42.892021 24232 catalog_manager.cc:4810] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): delete failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 already in progress: copying tablet
I20240502 21:27:42.895879 24649 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:27:42.897042 30866 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:42.906766 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:42.908000 26740 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:42.924114 30630 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:42.930060 24519 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:42.931202 26345 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:42.934105 24242 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 10 to 11, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:42.940215 24649 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:42.948796 24520 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:42.949770 30630 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:42.952153 30480 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:42.952795 24520 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:42.956301 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82fb425a5430426f8030ba5497736b11 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:42.958676 24244 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:42.964566 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fb425a5430426f8030ba5497736b11 with delete_type TABLET_DATA_TOMBSTONED (TS 1769ee000eb24298a367a0de1209e6e0 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:42.965749 26345 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:42.966897 30630 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:42.967707 30747 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:42.968139 30747 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Bootstrap complete.
I20240502 21:27:42.968426 30747 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.632s user 0.058s sys 0.002s
I20240502 21:27:42.968703 26344 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:27:42.969110 30747 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:42.969318 30747 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: LEARNER
I20240502 21:27:42.969497 30747 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:27:42.970100 30747 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:42.972564 24651 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:27:43.000615 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 427345e9a65d417c84a773ac1925a18a-98b5668e0d504db5a7874f041c14dbfa received from {username='slave'} at 127.16.122.3:36233
I20240502 21:27:43.000751 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 427345e9a65d417c84a773ac1925a18a-98b5668e0d504db5a7874f041c14dbfa on tablet 98b5668e0d504db5a7874f041c14dbfa with peer 427345e9a65d417c84a773ac1925a18a
I20240502 21:27:43.005250 30888 tablet_replica.cc:330] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:43.005682 30888 raft_consensus.cc:2229] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:43.006021 30888 pending_rounds.cc:62] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Trying to abort 1 pending ops.
I20240502 21:27:43.006088 30888 pending_rounds.cc:68] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Aborting op as it isn't in flight: id { term: 1 index: 11 } timestamp: 7023350836668108800 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "82fb425a5430426f8030ba5497736b11" old_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } } new_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } } }
I20240502 21:27:43.006811 30888 raft_consensus.cc:2894] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Aborting config change with OpId 1.11: Aborted: Op aborted
I20240502 21:27:43.006953 30888 raft_consensus.cc:2866] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:43.007068 30888 raft_consensus.cc:2258] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:43.008293 30727 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:43.008774 30727 tablet_bootstrap.cc:492] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: Bootstrap complete.
I20240502 21:27:43.009073 30727 ts_tablet_manager.cc:1367] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: Time spent bootstrapping tablet: real 0.808s user 0.048s sys 0.012s
I20240502 21:27:43.012526 30888 ts_tablet_manager.cc:1875] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:43.024544 30727 raft_consensus.cc:348] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:43.028664 30727 raft_consensus.cc:729] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b77f8f52372241d483f457389fda0f93, State: Initialized, Role: LEARNER
I20240502 21:27:43.028932 30727 consensus_queue.cc:260] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:43.031100 30727 ts_tablet_manager.cc:1398] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93: Time spent starting tablet: real 0.022s user 0.003s sys 0.001s
I20240502 21:27:43.031909 24673 tablet_copy_service.cc:342] P 427345e9a65d417c84a773ac1925a18a: Request end of tablet copy session b77f8f52372241d483f457389fda0f93-049bec57d2474c328d2e26e16a4d33a9 received from {username='slave'} at 127.16.122.5:36629
I20240502 21:27:43.032054 24673 tablet_copy_service.cc:433] P 427345e9a65d417c84a773ac1925a18a: ending tablet copy session b77f8f52372241d483f457389fda0f93-049bec57d2474c328d2e26e16a4d33a9 on tablet 049bec57d2474c328d2e26e16a4d33a9 with peer b77f8f52372241d483f457389fda0f93
I20240502 21:27:43.039395 30888 ts_tablet_manager.cc:1888] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 21:27:43.039525 30888 log.cc:1196] T 82fb425a5430426f8030ba5497736b11 P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/82fb425a5430426f8030ba5497736b11
I20240502 21:27:43.040616 24232 catalog_manager.cc:4830] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): tablet 82fb425a5430426f8030ba5497736b11 (table rebalance_test_table_7 [id=8931a08fc8114700bc668777de02e51c]) successfully deleted
I20240502 21:27:43.071821 30602 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:43.072326 30602 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:27:43.072671 30602 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.640s user 0.087s sys 0.041s
I20240502 21:27:43.073789 30602 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:43.074144 30602 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:27:43.074398 30602 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: NON_VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:27:43.075847 30602 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:43.076814 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-7bf18ffb96d04d5e9ec8ee9563812ba0 received from {username='slave'} at 127.16.122.2:59747
I20240502 21:27:43.076936 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 7f8bb40905da4891b461220b95c66832-7bf18ffb96d04d5e9ec8ee9563812ba0 on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:27:43.082940 24500 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 10)) from {username='slave'} at 127.0.0.1:57654
I20240502 21:27:43.083360 30870 tablet_replica.cc:330] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: stopping tablet replica
I20240502 21:27:43.083640 30870 raft_consensus.cc:2229] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 21:27:43.083822 30870 raft_consensus.cc:2258] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 21:27:43.086035 30870 ts_tablet_manager.cc:1875] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:43.096132 30870 ts_tablet_manager.cc:1888] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:43.096248 30870 log.cc:1196] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 7f8bb40905da4891b461220b95c66832: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-1/wal/wals/7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:43.097020 24232 catalog_manager.cc:4830] TS 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
I20240502 21:27:43.120193 26609 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
W20240502 21:27:43.138155 26543 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): Couldn't send request to peer e856847c3f494e6aa57f9cecf117ce4d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:43.139689 26344 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 21:27:43.140625 24778 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 21:27:43.142915 30758 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.143415 30937 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.148650 30937 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:27:43.167008 24244 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 10 to 11, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:43.169847 26344 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:27:43.171037 24778 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:27:43.269452 24384 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:43.272239 24649 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 21:27:43.272914 24323 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:43.273039 26741 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:43.273795 30866 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.274518 30866 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.278249 30438 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:43.283828 26741 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:43.287225 24649 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:27:43.288408 24244 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 10 to 11, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:43.331768 24649 raft_consensus.cc:1203] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 21:27:43.337600 29224 raft_consensus.cc:1050] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 to VOTER
I20240502 21:27:43.338519 29224 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:27:43.341269 24649 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:43.341279 26609 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:43.341823 29224 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:43.342119 30388 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:43.344928 29224 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:43.346678 26609 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:43.350404 24649 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:27:43.351040 24242 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 13 to 14, 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:43.355326 26344 raft_consensus.cc:1203] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240502 21:27:43.404395 24718 connection.cc:535] client connection to 127.16.122.6:41751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.404724 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.404855 26543 connection.cc:535] client connection to 127.16.122.6:41751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.404891 26277 connection.cc:535] client connection to 127.16.122.6:41751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.405079 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.405114 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.405565 24587 connection.cc:535] client connection to 127.16.122.6:41751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.405797 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.406297 24456 connection.cc:535] client connection to 127.16.122.6:41751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.406955 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.410216 24323 connection.cc:535] client connection to 127.16.122.6:41751 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.410566 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:43.430539 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.430697 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.434056 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.436005 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.436553 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:43.436789 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.436890 24323 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.437117 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.437247 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.437340 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.437413 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:43.438911 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:43.689838 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.704497 24632 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:43.720541 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 21:27:43.740633 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:43.773555 24500 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:43.774621 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:43.786338 30438 raft_consensus.cc:1050] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 427345e9a65d417c84a773ac1925a18a to VOTER
I20240502 21:27:43.787312 30438 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:43.825908 31083 ts_tablet_manager.cc:902] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Initiating tablet copy from peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
I20240502 21:27:43.828864 31084 ts_tablet_manager.cc:902] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:43.833391 31084 tablet_copy_client.cc:308] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:27:43.834229 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 1398472b92c842f285b8c7cc43cc1527 ({username='slave'} at 127.16.122.4:51127)
I20240502 21:27:43.834448 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 from peer 1398472b92c842f285b8c7cc43cc1527 at {username='slave'} at 127.16.122.4:51127: session id = 1398472b92c842f285b8c7cc43cc1527-7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:43.838308 24408 tablet_copy_source_session.cc:205] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:43.840097 31084 ts_tablet_manager.cc:1875] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:43.845876 26741 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:43.846609 26609 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:43.847741 29224 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.848284 29224 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.849222 31023 raft_consensus.cc:1050] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a: attempting to promote NON_VOTER b77f8f52372241d483f457389fda0f93 to VOTER
I20240502 21:27:43.849890 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:43.849928 24649 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:43.850368 31023 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
I20240502 21:27:43.857262 30438 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.858309 26612 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:43.859057 31023 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:43.863062 26344 raft_consensus.cc:1261] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 [term 1 LEARNER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:27:43.863826 30630 consensus_queue.cc:1035] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:27:43.866039 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:43.882591 29224 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:43.883268 24651 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:43.888691 24240 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:43.892640 26609 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:43.898651 26740 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 427345e9a65d417c84a773ac1925a18a (127.16.122.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:43.900357 30630 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:43.905647 26345 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:43.906584 26609 raft_consensus.cc:2932] T 049bec57d2474c328d2e26e16a4d33a9 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:43.907516 26325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:43.913017 24248 catalog_manager.cc:5484] T 049bec57d2474c328d2e26e16a4d33a9 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 10 to 11, b77f8f52372241d483f457389fda0f93 (127.16.122.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } }
I20240502 21:27:43.937399 31084 ts_tablet_manager.cc:1888] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:27:43.951869 31084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:43.967026 31083 tablet_copy_client.cc:308] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.7:36311
I20240502 21:27:43.968451 26632 tablet_copy_service.cc:140] P 9b0952b24bd8400c9940dbd5e28417b4: Received BeginTabletCopySession request for tablet 2b6a4a74e0544e0e99e734a145c91d3b from peer e856847c3f494e6aa57f9cecf117ce4d ({username='slave'} at 127.16.122.1:36393)
I20240502 21:27:43.968729 26632 tablet_copy_service.cc:161] P 9b0952b24bd8400c9940dbd5e28417b4: Beginning new tablet copy session on tablet 2b6a4a74e0544e0e99e734a145c91d3b from peer e856847c3f494e6aa57f9cecf117ce4d at {username='slave'} at 127.16.122.1:36393: session id = e856847c3f494e6aa57f9cecf117ce4d-2b6a4a74e0544e0e99e734a145c91d3b
I20240502 21:27:43.971191 24384 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:27:43.973098 24649 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:43.973111 26632 tablet_copy_source_session.cc:205] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:43.974023 26612 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:43.975322 30438 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:43.983578 30438 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:43.988530 30438 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:43.990911 31084 tablet_copy_client.cc:783] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:43.991102 26612 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:43.999735 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98b5668e0d504db5a7874f041c14dbfa with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:44.005371 24242 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.010082 24649 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:27:44.010835 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b5668e0d504db5a7874f041c14dbfa with delete_type TABLET_DATA_TOMBSTONED (TS 1769ee000eb24298a367a0de1209e6e0 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:44.019794 31083 ts_tablet_manager.cc:1875] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:27:44.025710 31084 tablet_copy_client.cc:647] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.035591 31084 tablet_copy_client.cc:523] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:44.041232 31084 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:27:44.051523 31128 tablet_replica.cc:330] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:44.052381 31128 raft_consensus.cc:2229] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:44.052964 31128 raft_consensus.cc:2258] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:44.055037 31083 ts_tablet_manager.cc:1888] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.9
I20240502 21:27:44.057089 31128 ts_tablet_manager.cc:1875] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:44.065745 31128 ts_tablet_manager.cc:1888] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 21:27:44.065995 31128 log.cc:1196] T 98b5668e0d504db5a7874f041c14dbfa P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/98b5668e0d504db5a7874f041c14dbfa
I20240502 21:27:44.067063 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6a4a74e0544e0e99e734a145c91d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.067500 24232 catalog_manager.cc:4830] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): tablet 98b5668e0d504db5a7874f041c14dbfa (table rebalance_test_table_9 [id=6fefd52238f245dfaf693d8dd980f834]) successfully deleted
I20240502 21:27:44.097142 31083 tablet_copy_client.cc:783] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.104449 31083 tablet_copy_client.cc:647] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.113755 31083 tablet_copy_client.cc:523] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:44.123818 31023 raft_consensus.cc:482] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 836775e329f148e5908a1afc54f06428)
I20240502 21:27:44.124060 31023 raft_consensus.cc:504] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:44.125099 31023 leader_election.cc:288] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:27:44.127934 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "427345e9a65d417c84a773ac1925a18a" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "7f8bb40905da4891b461220b95c66832" is_pre_election: true
W20240502 21:27:44.130049 24587 leader_election.cc:334] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:44.130174 24587 leader_election.cc:302] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a; no voters: 7f8bb40905da4891b461220b95c66832, 836775e329f148e5908a1afc54f06428
I20240502 21:27:44.130463 31023 raft_consensus.cc:2726] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:44.131958 30843 raft_consensus.cc:482] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 836775e329f148e5908a1afc54f06428)
I20240502 21:27:44.132179 30843 raft_consensus.cc:504] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:44.133041 30843 leader_election.cc:288] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751)
W20240502 21:27:44.135421 24456 leader_election.cc:334] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:44.136719 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a" is_pre_election: true
I20240502 21:27:44.136914 24649 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 2.
I20240502 21:27:44.139084 24457 leader_election.cc:302] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters: 836775e329f148e5908a1afc54f06428
I20240502 21:27:44.139724 30843 raft_consensus.cc:2781] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:27:44.139806 30843 raft_consensus.cc:482] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 836775e329f148e5908a1afc54f06428)
I20240502 21:27:44.139873 30843 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:44.144610 30843 raft_consensus.cc:504] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:44.146427 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a9b14a7ff04ddab0d1e1a8e9fcb9b0" candidate_uuid: "7f8bb40905da4891b461220b95c66832" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "427345e9a65d417c84a773ac1925a18a"
I20240502 21:27:44.146683 24649 raft_consensus.cc:3037] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:44.150699 31084 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:44.151203 31084 tablet_bootstrap.cc:492] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap complete.
I20240502 21:27:44.151594 31084 ts_tablet_manager.cc:1367] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.111s user 0.075s sys 0.004s
I20240502 21:27:44.152427 31084 raft_consensus.cc:348] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:44.152737 31084 raft_consensus.cc:729] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: LEARNER
I20240502 21:27:44.152954 31084 consensus_queue.cc:260] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:27:44.153709 31084 ts_tablet_manager.cc:1398] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:44.154491 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1398472b92c842f285b8c7cc43cc1527-7bf18ffb96d04d5e9ec8ee9563812ba0 received from {username='slave'} at 127.16.122.4:51127
I20240502 21:27:44.154608 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1398472b92c842f285b8c7cc43cc1527-7bf18ffb96d04d5e9ec8ee9563812ba0 on tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with peer 1398472b92c842f285b8c7cc43cc1527
W20240502 21:27:44.155103 24456 leader_election.cc:334] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:44.153160 24649 raft_consensus.cc:2454] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8bb40905da4891b461220b95c66832 in term 3.
I20240502 21:27:44.155673 24457 leader_election.cc:302] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 427345e9a65d417c84a773ac1925a18a, 7f8bb40905da4891b461220b95c66832; no voters: 836775e329f148e5908a1afc54f06428
I20240502 21:27:44.155961 30544 raft_consensus.cc:2781] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:44.156069 30544 raft_consensus.cc:686] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 3 LEADER]: Becoming Leader. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Running, Role: LEADER
I20240502 21:27:44.156324 30544 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:27:44.157711 30843 leader_election.cc:288] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 election: Requested vote from peers 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427), 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751)
I20240502 21:27:44.160723 24242 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: term changed from 2 to 3, leader changed from 836775e329f148e5908a1afc54f06428 (127.16.122.6) to 7f8bb40905da4891b461220b95c66832 (127.16.122.2). New cstate: current_term: 3 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:44.167932 31083 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
W20240502 21:27:44.237562 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:44.265908 31083 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:44.266631 31083 tablet_bootstrap.cc:492] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap complete.
I20240502 21:27:44.267009 31083 ts_tablet_manager.cc:1367] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.099s user 0.054s sys 0.018s
I20240502 21:27:44.268468 31083 raft_consensus.cc:348] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
I20240502 21:27:44.268884 31083 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: LEARNER
I20240502 21:27:44.269191 31083 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
I20240502 21:27:44.270109 31083 ts_tablet_manager.cc:1398] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:44.271215 26632 tablet_copy_service.cc:342] P 9b0952b24bd8400c9940dbd5e28417b4: Request end of tablet copy session e856847c3f494e6aa57f9cecf117ce4d-2b6a4a74e0544e0e99e734a145c91d3b received from {username='slave'} at 127.16.122.1:36393
I20240502 21:27:44.271476 26632 tablet_copy_service.cc:433] P 9b0952b24bd8400c9940dbd5e28417b4: ending tablet copy session e856847c3f494e6aa57f9cecf117ce4d-2b6a4a74e0544e0e99e734a145c91d3b on tablet 2b6a4a74e0544e0e99e734a145c91d3b with peer e856847c3f494e6aa57f9cecf117ce4d
I20240502 21:27:44.295738 24778 raft_consensus.cc:1203] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:27:44.486513 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:44.508795 24632 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:44.522796 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.524039 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:44.554162 24649 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 3 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 21:27:44.554826 30843 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:27:44.583005 26325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:44.588939 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 21:27:44.604969 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.649559 24500 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:44.660164 24384 raft_consensus.cc:1203] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240502 21:27:44.725219 30866 raft_consensus.cc:1050] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1398472b92c842f285b8c7cc43cc1527 to VOTER
I20240502 21:27:44.730291 30866 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:44.743759 24783 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:44.744607 29224 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 21:27:44.752380 24323 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.754781 26741 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:44.755621 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:44.759341 24651 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:27:44.759990 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:44.766659 26797 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 21:27:14.810935 26804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:14.811359 26804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:14.825574 26804 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:17.637522 26804 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:27:19.416124 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.416489 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:19.416867 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:19.417060 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:19.417272 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:19.417594 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:19.417876 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.418056 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.418185 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:19.418329 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:19.418483 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:19.418701 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.418834 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:19.419046 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:19.738710 26804 rebalancer_tool.cc:1074] tablet e847a796029d4edea79694f5402a291a: '7f8bb40905da4891b461220b95c66832' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:19.897763 26804 rebalancer_tool.cc:1074] tablet 0fc717a1b12e4380a4107e6deae2b837: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:20.280712 26804 rebalancer_tool.cc:1074] tablet 9d8a25580bc74004b0e31c34f841d2cc: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:20.665992 26804 rebalancer_tool.cc:1074] tablet 049bec57d2474c328d2e26e16a4d33a9: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:20.808102 26804 rebalancer_tool.cc:1074] tablet 2b6a4a74e0544e0e99e734a145c91d3b: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:21.117116 26804 rebalancer_tool.cc:1074] tablet ff7e019fa82d4648afaf130d3f6343d6: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:21.483076 26804 rebalancer_tool.cc:1074] tablet 566929e603d54be4bf931413f0fa1d7d: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:21.929199 26804 rebalancer_tool.cc:1084] tablet ac35b080037646b4b8b34c0eab2de4e6: '1398472b92c842f285b8c7cc43cc1527' -> '9b0952b24bd8400c9940dbd5e28417b4' move ignored: Illegal state: tablet ac35b080037646b4b8b34c0eab2de4e6: replica 9b0952b24bd8400c9940dbd5e28417b4 is already present
I20240502 21:27:21.929419 26804 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:26.948024 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.948175 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.949936 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.950086 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:26.950222 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.950814 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:26.950975 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:26.951097 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:26.951222 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.951340 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:26.952320 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:27.496825 26804 rebalancer_tool.cc:1074] tablet 6c4aa176fd994cf0b55f304cf9d06fe0: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:28.146159 26804 rebalancer_tool.cc:1074] tablet 98b5668e0d504db5a7874f041c14dbfa: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:28.441987 26804 rebalancer_tool.cc:1074] tablet d0ba1af1d3624d6387bf892873f3f67e: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:28.788061 26804 rebalancer_tool.cc:1084] tablet 53c1992a21224bac883da474f143c1d7: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move ignored: Illegal state: tablet 53c1992a21224bac883da474f143c1d7: replica b77f8f52372241d483f457389fda0f93 is already present
I20240502 21:27:28.788789 26804 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:32.740118 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:32.748090 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 427345e9a65d417c84a773ac1925a18a
W20240502 21:27:32.748289 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server b77f8f52372241d483f457389fda0f93 to server 836775e329f148e5908a1afc54f06428
I20240502 21:27:32.881716 26804 rebalancer_tool.cc:1084] tablet 5c6e7eba855e48c0b0e18fa363ea01d4: '427345e9a65d417c84a773ac1925a18a' -> '836775e329f148e5908a1afc54f06428' move ignored: Illegal state: tablet 5c6e7eba855e48c0b0e18fa363ea01d4: replica 836775e329f148e5908a1afc54f06428 is already present
I20240502 21:27:32.881899 26804 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:35.188758 26804 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 427345e9a65d417c84a773ac1925a18a
I20240502 21:27:35.436925 26804 rebalancer_tool.cc:1074] tablet bdf23b7a397441b5947ce6babe3e8f62: '427345e9a65d417c84a773ac1925a18a' -> '' move scheduled
I20240502 21:27:35.690364 26804 rebalancer_tool.cc:1074] tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0: '1398472b92c842f285b8c7cc43cc1527' -> '7f8bb40905da4891b461220b95c66832' move scheduled
I20240502 21:27:36.094170 26804 rebalancer_tool.cc:1074] tablet 8f912eb05c574368909e0b3a8edf058d: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:36.229832 26804 rebalancer_tool.cc:1074] tablet 82fb425a5430426f8030ba5497736b11: '7f8bb40905da4891b461220b95c66832' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:36.433655 26804 rebalancer_tool.cc:1074] tablet 9963dcea478749e0a6e076fac6537fc4: 'b77f8f52372241d483f457389fda0f93' -> '427345e9a65d417c84a773ac1925a18a' move scheduled
I20240502 21:27:36.963259 26804 rebalancer_tool.cc:1136] tablet bdf23b7a397441b5947ce6babe3e8f62: '427345e9a65d417c84a773ac1925a18a' -> '' move completed: OK
I20240502 21:27:37.883067 26804 rebalancer_tool.cc:1136] tablet 98b5668e0d504db5a7874f041c14dbfa: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:38.159092 26804 rebalancer_tool.cc:1136] tablet 6c4aa176fd994cf0b55f304cf9d06fe0: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move completed: Incomplete: tablet 6c4aa176fd994cf0b55f304cf9d06fe0, TS 1398472b92c842f285b8c7cc43cc1527 -> TS 836775e329f148e5908a1afc54f06428 move failed, target replica disappeared
I20240502 21:27:38.324357 26804 rebalancer_tool.cc:1136] tablet 0fc717a1b12e4380a4107e6deae2b837: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:38.443192 26804 rebalancer_tool.cc:1136] tablet e847a796029d4edea79694f5402a291a: '7f8bb40905da4891b461220b95c66832' -> 'b77f8f52372241d483f457389fda0f93' move completed: OK
I20240502 21:27:39.351967 26804 rebalancer_tool.cc:1136] tablet 9d8a25580bc74004b0e31c34f841d2cc: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: Incomplete: tablet 9d8a25580bc74004b0e31c34f841d2cc, TS 427345e9a65d417c84a773ac1925a18a -> TS 9b0952b24bd8400c9940dbd5e28417b4 move failed, target replica disappeared
I20240502 21:27:40.017345 26804 rebalancer_tool.cc:1136] tablet 049bec57d2474c328d2e26e16a4d33a9: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move completed: OK
I20240502 21:27:40.415227 26804 rebalancer_tool.cc:1136] tablet 2b6a4a74e0544e0e99e734a145c91d3b: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:40.630471 26804 rebalancer_tool.cc:1136] tablet ff7e019fa82d4648afaf130d3f6343d6: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move completed: OK
I20240502 21:27:40.770486 26804 rebalancer_tool.cc:1136] tablet 9963dcea478749e0a6e076fac6537fc4: 'b77f8f52372241d483f457389fda0f93' -> '427345e9a65d417c84a773ac1925a18a' move completed: OK
I20240502 21:27:40.812567 26804 rebalancer_tool.cc:1136] tablet 82fb425a5430426f8030ba5497736b11: '7f8bb40905da4891b461220b95c66832' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:40.934648 26804 rebalancer_tool.cc:1136] tablet 8f912eb05c574368909e0b3a8edf058d: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move completed: OK
I20240502 21:27:40.985617 26804 rebalancer_tool.cc:1136] tablet d0ba1af1d3624d6387bf892873f3f67e: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:41.039825 26804 rebalancer_tool.cc:1136] tablet 566929e603d54be4bf931413f0fa1d7d: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:41.103296 26804 rebalancer_tool.cc:1136] tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0: '1398472b92c842f285b8c7cc43cc1527' -> '7f8bb40905da4891b461220b95c66832' move completed: OK
I20240502 21:27:43.465413 26804 rebalancer_tool.cc:1084] tablet 2b6a4a74e0544e0e99e734a145c91d3b: '9b0952b24bd8400c9940dbd5e28417b4' -> 'e856847c3f494e6aa57f9cecf117ce4d' move ignored: Illegal state: tablet 2b6a4a74e0544e0e99e734a145c91d3b: replica e856847c3f494e6aa57f9cecf117ce4d is already present
I20240502 21:27:43.465580 26804 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): unacceptable health status UNAVAILABLE
I20240502 21:27:44.781939 29224 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.786324 26741 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.785856 24649 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
W20240502 21:27:44.787865 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:44.788813 24783 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.792408 24244 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 11 to 12, 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:44.803937 24384 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:44.806159 24783 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:27:44.807226 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:44.808709 24649 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240502 21:27:44.809288 24323 consensus_peers.cc:485] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.809469 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:44.813036 31247 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.813705 24649 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.817786 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:44.819128 24783 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.820472 24240 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 12 to 13, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:44.825068 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with delete_type TABLET_DATA_TOMBSTONED (TS 1769ee000eb24298a367a0de1209e6e0 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49940
I20240502 21:27:44.826259 24384 consensus_queue.cc:237] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:27:44.828789 24783 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:44.830518 31247 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:44.832803 24651 raft_consensus.cc:1261] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:27:44.832924 31257 tablet_replica.cc:330] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: stopping tablet replica
I20240502 21:27:44.833159 31257 raft_consensus.cc:2229] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:44.833561 30438 consensus_queue.cc:1035] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:44.833878 31257 raft_consensus.cc:2258] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:44.834774 30438 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.835386 31257 ts_tablet_manager.cc:1875] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:44.838390 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:44.840499 24240 catalog_manager.cc:5484] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 13 to 14, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.841322 24651 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.841961 24783 raft_consensus.cc:2932] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } }
I20240502 21:27:44.842417 31257 ts_tablet_manager.cc:1888] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:27:44.842633 31257 log.cc:1196] T 7bf18ffb96d04d5e9ec8ee9563812ba0 P 1769ee000eb24298a367a0de1209e6e0: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-7/wal/wals/7bf18ffb96d04d5e9ec8ee9563812ba0
I20240502 21:27:44.843554 24232 catalog_manager.cc:4830] TS 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): tablet 7bf18ffb96d04d5e9ec8ee9563812ba0 (table rebalance_test_table_0 [id=63de16df705b4df0abcb6fcd022351ac]) successfully deleted
W20240502 21:27:44.850692 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:44.852126 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:45.019546 26795 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 21:27:14.625620 26801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:14.626116 26801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:14.639144 26801 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:18.135699 26801 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:27:20.558197 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:20.558624 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:20.558794 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:20.559078 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:20.559171 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:20.559270 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:20.559365 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:20.559502 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:20.559592 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:20.559688 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:20.559778 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:20.559870 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:20.559947 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:20.560052 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:20.560137 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:20.560230 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server b77f8f52372241d483f457389fda0f93
I20240502 21:27:20.757504 26801 rebalancer_tool.cc:1074] tablet ac35b080037646b4b8b34c0eab2de4e6: 'e856847c3f494e6aa57f9cecf117ce4d' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:20.969661 26801 rebalancer_tool.cc:1074] tablet 9d8a25580bc74004b0e31c34f841d2cc: '7f8bb40905da4891b461220b95c66832' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:21.256954 26801 rebalancer_tool.cc:1074] tablet 9963dcea478749e0a6e076fac6537fc4: '427345e9a65d417c84a773ac1925a18a' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:21.776672 26801 rebalancer_tool.cc:1074] tablet dc5c40e2bf0340958ecd3e7541209e95: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:22.195022 26801 rebalancer_tool.cc:1074] tablet 855b247f654d41c2bbb13abb25d35c1c: '1398472b92c842f285b8c7cc43cc1527' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:23.560880 26801 rebalancer_tool.cc:1074] tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0: '7f8bb40905da4891b461220b95c66832' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:24.624692 26801 rebalancer_tool.cc:1084] tablet adde18faca794600bb4aa24f0cf744e6: '427345e9a65d417c84a773ac1925a18a' -> 'b77f8f52372241d483f457389fda0f93' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 21:27:24.624866 26801 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:26.309582 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.309953 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:26.310220 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:26.311606 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:26.311738 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:26.311884 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:26.312052 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:26.312201 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server b77f8f52372241d483f457389fda0f93
I20240502 21:27:26.677964 26801 rebalancer_tool.cc:1074] tablet 566929e603d54be4bf931413f0fa1d7d: '1398472b92c842f285b8c7cc43cc1527' -> '' move scheduled
I20240502 21:27:26.811242 26801 rebalancer_tool.cc:1074] tablet 53c1992a21224bac883da474f143c1d7: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:27.981720 26801 rebalancer_tool.cc:1074] tablet 477d9df7d3a74b8584f3669be3256e00: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:28.306740 26801 rebalancer_tool.cc:1074] tablet 5dd91818e74e42acb276a3864753094d: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:28.657291 26801 rebalancer_tool.cc:1074] tablet bdf23b7a397441b5947ce6babe3e8f62: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:28.864728 26801 rebalancer_tool.cc:1074] tablet 2b6a4a74e0544e0e99e734a145c91d3b: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:29.462054 26801 rebalancer_tool.cc:1074] tablet e847a796029d4edea79694f5402a291a: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:30.060860 26801 rebalancer_tool.cc:1074] tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec: '7f8bb40905da4891b461220b95c66832' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:30.596268 26801 rebalancer_tool.cc:1084] tablet d0ba1af1d3624d6387bf892873f3f67e: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move ignored: Illegal state: tablet d0ba1af1d3624d6387bf892873f3f67e: replica 9b0952b24bd8400c9940dbd5e28417b4 is already present
I20240502 21:27:30.596442 26801 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:32.790860 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:32.791067 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server e856847c3f494e6aa57f9cecf117ce4d
W20240502 21:27:32.791141 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server b77f8f52372241d483f457389fda0f93 to server 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:33.074616 26801 rebalancer_tool.cc:1074] tablet 0fc717a1b12e4380a4107e6deae2b837: '427345e9a65d417c84a773ac1925a18a' -> '' move scheduled
I20240502 21:27:33.374073 26801 rebalancer_tool.cc:1084] tablet 6c4aa176fd994cf0b55f304cf9d06fe0: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move ignored: Illegal state: tablet 6c4aa176fd994cf0b55f304cf9d06fe0: replica 1769ee000eb24298a367a0de1209e6e0 is already present
I20240502 21:27:33.374217 26801 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:35.819700 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server b77f8f52372241d483f457389fda0f93 to server 427345e9a65d417c84a773ac1925a18a
W20240502 21:27:35.819918 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1398472b92c842f285b8c7cc43cc1527
I20240502 21:27:35.937655 26801 rebalancer_tool.cc:1074] tablet 7a35079a106c4cca94cf65ffa93aefbf: '9b0952b24bd8400c9940dbd5e28417b4' -> '427345e9a65d417c84a773ac1925a18a' move scheduled
I20240502 21:27:36.067431 26801 rebalancer_tool.cc:1084] tablet 8f912eb05c574368909e0b3a8edf058d: '7f8bb40905da4891b461220b95c66832' -> 'b77f8f52372241d483f457389fda0f93' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:36.067597 26801 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:38.354091 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1398472b92c842f285b8c7cc43cc1527
W20240502 21:27:38.354267 26801 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 836775e329f148e5908a1afc54f06428
I20240502 21:27:38.428164 26801 rebalancer_tool.cc:1136] tablet 0fc717a1b12e4380a4107e6deae2b837: '427345e9a65d417c84a773ac1925a18a' -> '' move completed: OK
I20240502 21:27:38.740167 26801 rebalancer_tool.cc:1136] tablet e847a796029d4edea79694f5402a291a: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:39.108594 26801 rebalancer_tool.cc:1136] tablet 7a35079a106c4cca94cf65ffa93aefbf: '9b0952b24bd8400c9940dbd5e28417b4' -> '427345e9a65d417c84a773ac1925a18a' move completed: OK
I20240502 21:27:39.447301 26801 rebalancer_tool.cc:1136] tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec: '7f8bb40905da4891b461220b95c66832' -> '836775e329f148e5908a1afc54f06428' move completed: OK
I20240502 21:27:39.841894 26801 rebalancer_tool.cc:1136] tablet 5dd91818e74e42acb276a3864753094d: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:40.081524 26801 rebalancer_tool.cc:1136] tablet 53c1992a21224bac883da474f143c1d7: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move completed: OK
I20240502 21:27:40.460774 26801 rebalancer_tool.cc:1136] tablet 566929e603d54be4bf931413f0fa1d7d: '1398472b92c842f285b8c7cc43cc1527' -> '' move completed: OK
I20240502 21:27:40.630237 26801 rebalancer_tool.cc:1136] tablet 9d8a25580bc74004b0e31c34f841d2cc: '7f8bb40905da4891b461220b95c66832' -> '836775e329f148e5908a1afc54f06428' move completed: OK
I20240502 21:27:40.725054 26801 rebalancer_tool.cc:1123] tablet bdf23b7a397441b5947ce6babe3e8f62: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move is abandoned: Illegal state: bdf23b7a397441b5947ce6babe3e8f62: source replica 1398472b92c842f285b8c7cc43cc1527 does not have REPLACE attribute set
I20240502 21:27:40.786909 26801 rebalancer_tool.cc:1136] tablet ac35b080037646b4b8b34c0eab2de4e6: 'e856847c3f494e6aa57f9cecf117ce4d' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:40.882140 26801 rebalancer_tool.cc:1136] tablet dc5c40e2bf0340958ecd3e7541209e95: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:40.967736 26801 rebalancer_tool.cc:1123] tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0: '7f8bb40905da4891b461220b95c66832' -> '836775e329f148e5908a1afc54f06428' move is abandoned: Illegal state: a8a9b14a7ff04ddab0d1e1a8e9fcb9b0: source replica 7f8bb40905da4891b461220b95c66832 does not have REPLACE attribute set
I20240502 21:27:41.048313 26801 rebalancer_tool.cc:1136] tablet 9963dcea478749e0a6e076fac6537fc4: '427345e9a65d417c84a773ac1925a18a' -> 'b77f8f52372241d483f457389fda0f93' move completed: Incomplete: tablet 9963dcea478749e0a6e076fac6537fc4, TS 427345e9a65d417c84a773ac1925a18a -> TS b77f8f52372241d483f457389fda0f93 move failed, target replica disappeared
I20240502 21:27:41.145262 26801 rebalancer_tool.cc:1136] tablet 477d9df7d3a74b8584f3669be3256e00: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:41.228155 26801 rebalancer_tool.cc:1136] tablet 855b247f654d41c2bbb13abb25d35c1c: '1398472b92c842f285b8c7cc43cc1527' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:41.357285 26801 rebalancer_tool.cc:1074] tablet 98b5668e0d504db5a7874f041c14dbfa: '1769ee000eb24298a367a0de1209e6e0' -> '427345e9a65d417c84a773ac1925a18a' move scheduled
I20240502 21:27:41.591645 26801 rebalancer_tool.cc:1136] tablet 2b6a4a74e0544e0e99e734a145c91d3b: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move completed: OK
I20240502 21:27:44.025986 26801 rebalancer_tool.cc:1136] tablet 98b5668e0d504db5a7874f041c14dbfa: '1769ee000eb24298a367a0de1209e6e0' -> '427345e9a65d417c84a773ac1925a18a' move completed: OK
Illegal state: tablet server 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): unacceptable health status UNAVAILABLE
I20240502 21:27:45.142367 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:45.165534 24632 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:45.197427 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:45.206274 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:45.215533 24500 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:45.235898 26325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:45.251268 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:45.258673 26592 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:45.265075 30941 raft_consensus.cc:1050] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: attempting to promote NON_VOTER e856847c3f494e6aa57f9cecf117ce4d to VOTER
I20240502 21:27:45.266089 30941 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:27:45.280346 24384 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 LEARNER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:27:45.280630 26344 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:27:45.281009 31158 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:45.283178 31158 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:45.289155 24783 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:27:45.291178 30941 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:27:45.298075 31158 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:45.303705 26344 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:45.310736 24246 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 11 to 12, e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
W20240502 21:27:45.314522 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:45.316486 24783 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:45.328562 24384 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:45.366485 26612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59270
I20240502 21:27:45.366804 26612 raft_consensus.cc:595] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Received request to transfer leadership
W20240502 21:27:45.378717 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:45.567041 26799 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 21:27:14.550827 26800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:14.551277 26800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:14.564721 26800 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:18.139885 26800 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:27:19.839267 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:19.840694 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:19.840947 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.841383 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:19.844009 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:19.844228 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:19.844345 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:19.844455 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.844554 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.844676 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:19.844781 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:19.844981 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:19.845110 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:19.845225 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
I20240502 21:27:20.094703 26800 rebalancer_tool.cc:1084] tablet 0fc717a1b12e4380a4107e6deae2b837: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move ignored: Illegal state: tablet 0fc717a1b12e4380a4107e6deae2b837: replica 1769ee000eb24298a367a0de1209e6e0 is already present
I20240502 21:27:20.095188 26800 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:21.364859 27557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.16.122.2:43159, user_credentials={real_user=slave}} blocked reactor thread for 64024.7us
W20240502 21:27:23.633700 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:23.633847 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:23.634020 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:23.634209 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:23.673352 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:23.673952 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:23.674149 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:23.674269 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:23.674412 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:23.674592 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:23.674695 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:23.674846 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:23.674957 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:23.675072 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:23.675180 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:23.675299 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:27:23.961479 26800 rebalancer_tool.cc:1074] tablet ac35b080037646b4b8b34c0eab2de4e6: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:25.636600 26800 rebalancer_tool.cc:1074] tablet 7a35079a106c4cca94cf65ffa93aefbf: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:25.817778 26800 rebalancer_tool.cc:1084] tablet 049bec57d2474c328d2e26e16a4d33a9: '1398472b92c842f285b8c7cc43cc1527' -> '9b0952b24bd8400c9940dbd5e28417b4' move ignored: Not found: Peer with uuid 1398472b92c842f285b8c7cc43cc1527 not found in consensus config
I20240502 21:27:25.818006 26800 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:27.932193 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:27.932600 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:27.932760 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:27.932917 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:27.933043 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:27.933166 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:27.933269 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:27.933439 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:27.933583 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:28.322933 26800 rebalancer_tool.cc:1084] tablet d0ba1af1d3624d6387bf892873f3f67e: '7f8bb40905da4891b461220b95c66832' -> '9b0952b24bd8400c9940dbd5e28417b4' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:28.323053 26800 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:32.271947 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 427345e9a65d417c84a773ac1925a18a
W20240502 21:27:32.272222 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 836775e329f148e5908a1afc54f06428
I20240502 21:27:32.566171 26800 rebalancer_tool.cc:1074] tablet 2e5a87719c2544a5b0cb591ed9f46df8: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:32.771067 26800 rebalancer_tool.cc:1074] tablet dc5c40e2bf0340958ecd3e7541209e95: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:33.303609 26800 rebalancer_tool.cc:1074] tablet ff7e019fa82d4648afaf130d3f6343d6: '7f8bb40905da4891b461220b95c66832' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:33.479380 26800 rebalancer_tool.cc:1084] tablet 6c4aa176fd994cf0b55f304cf9d06fe0: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move ignored: Illegal state: tablet 6c4aa176fd994cf0b55f304cf9d06fe0: replica 1769ee000eb24298a367a0de1209e6e0 is already present
I20240502 21:27:33.479560 26800 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:36.315238 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server e856847c3f494e6aa57f9cecf117ce4d
I20240502 21:27:36.497269 26800 rebalancer_tool.cc:1074] tablet 566929e603d54be4bf931413f0fa1d7d: 'b77f8f52372241d483f457389fda0f93' -> '' move scheduled
I20240502 21:27:36.738247 26800 rebalancer_tool.cc:1074] tablet 9d8a25580bc74004b0e31c34f841d2cc: '9b0952b24bd8400c9940dbd5e28417b4' -> '427345e9a65d417c84a773ac1925a18a' move scheduled
I20240502 21:27:36.892055 26800 rebalancer_tool.cc:1074] tablet 477d9df7d3a74b8584f3669be3256e00: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:37.182406 26800 rebalancer_tool.cc:1074] tablet 0fc717a1b12e4380a4107e6deae2b837: '7f8bb40905da4891b461220b95c66832' -> '' move scheduled
I20240502 21:27:37.699442 26800 rebalancer_tool.cc:1084] tablet 8f912eb05c574368909e0b3a8edf058d: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move ignored: Illegal state: tablet 8f912eb05c574368909e0b3a8edf058d: replica b77f8f52372241d483f457389fda0f93 is already present
I20240502 21:27:37.699615 26800 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:40.435133 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 836775e329f148e5908a1afc54f06428 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:40.435268 26800 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 427345e9a65d417c84a773ac1925a18a
I20240502 21:27:40.678107 26800 rebalancer_tool.cc:1074] tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: '1769ee000eb24298a367a0de1209e6e0' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:40.858268 26800 rebalancer_tool.cc:1074] tablet 82fb425a5430426f8030ba5497736b11: '1769ee000eb24298a367a0de1209e6e0' -> '7f8bb40905da4891b461220b95c66832' move scheduled
I20240502 21:27:41.126644 26800 rebalancer_tool.cc:1136] tablet 0fc717a1b12e4380a4107e6deae2b837: '7f8bb40905da4891b461220b95c66832' -> '' move completed: OK
I20240502 21:27:41.251502 26800 rebalancer_tool.cc:1136] tablet 477d9df7d3a74b8584f3669be3256e00: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move completed: Incomplete: tablet 477d9df7d3a74b8584f3669be3256e00, TS e856847c3f494e6aa57f9cecf117ce4d -> TS 1769ee000eb24298a367a0de1209e6e0 move failed, target replica disappeared
I20240502 21:27:41.463369 26800 rebalancer_tool.cc:1123] tablet 7a35079a106c4cca94cf65ffa93aefbf: '427345e9a65d417c84a773ac1925a18a' -> '1769ee000eb24298a367a0de1209e6e0' move is abandoned: Illegal state: 7a35079a106c4cca94cf65ffa93aefbf: source replica 427345e9a65d417c84a773ac1925a18a does not have REPLACE attribute set
I20240502 21:27:41.550029 26800 rebalancer_tool.cc:1136] tablet ac35b080037646b4b8b34c0eab2de4e6: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move completed: Incomplete: tablet ac35b080037646b4b8b34c0eab2de4e6, TS e856847c3f494e6aa57f9cecf117ce4d -> TS b77f8f52372241d483f457389fda0f93 move failed, target replica disappeared
I20240502 21:27:41.852593 26800 rebalancer_tool.cc:1136] tablet 2e5a87719c2544a5b0cb591ed9f46df8: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move completed: OK
I20240502 21:27:41.946554 26800 rebalancer_tool.cc:1136] tablet ff7e019fa82d4648afaf130d3f6343d6: '7f8bb40905da4891b461220b95c66832' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:42.201126 26800 rebalancer_tool.cc:1123] tablet 9d8a25580bc74004b0e31c34f841d2cc: '9b0952b24bd8400c9940dbd5e28417b4' -> '427345e9a65d417c84a773ac1925a18a' move is abandoned: Illegal state: 9d8a25580bc74004b0e31c34f841d2cc: source replica 9b0952b24bd8400c9940dbd5e28417b4 does not have REPLACE attribute set
I20240502 21:27:42.467725 26800 rebalancer_tool.cc:1136] tablet 566929e603d54be4bf931413f0fa1d7d: 'b77f8f52372241d483f457389fda0f93' -> '' move completed: OK
I20240502 21:27:43.013521 26800 rebalancer_tool.cc:1136] tablet dc5c40e2bf0340958ecd3e7541209e95: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:43.086397 26800 rebalancer_tool.cc:1136] tablet 82fb425a5430426f8030ba5497736b11: '1769ee000eb24298a367a0de1209e6e0' -> '7f8bb40905da4891b461220b95c66832' move completed: OK
I20240502 21:27:44.961714 26800 rebalancer_tool.cc:1136] tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: '1769ee000eb24298a367a0de1209e6e0' -> '836775e329f148e5908a1afc54f06428' move completed: Incomplete: tablet 7bf18ffb96d04d5e9ec8ee9563812ba0, TS 1769ee000eb24298a367a0de1209e6e0 -> TS 836775e329f148e5908a1afc54f06428 move failed, target replica disappeared
Illegal state: tablet server 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): unacceptable health status UNAVAILABLE
I20240502 21:27:45.584091 26612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59286
I20240502 21:27:45.584304 26612 raft_consensus.cc:595] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Received request to transfer leadership
W20240502 21:27:45.660871 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:45.670764 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:45.679997 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:45.705827 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:45.718107 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:45.734021 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:45.761143 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:45.803639 26612 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b"
dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59312
I20240502 21:27:45.803906 26612 raft_consensus.cc:595] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Received request to transfer leadership
I20240502 21:27:45.851871 31350 raft_consensus.cc:979] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: : Instructing follower b77f8f52372241d483f457389fda0f93 to start an election
I20240502 21:27:45.852077 31350 raft_consensus.cc:1067] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Signalling peer b77f8f52372241d483f457389fda0f93 to start an election
I20240502 21:27:45.852813 26344 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b"
dest_uuid: "b77f8f52372241d483f457389fda0f93"
from {username='slave'} at 127.16.122.7:42461
I20240502 21:27:45.852996 26344 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:45.853101 26344 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:45.855073 26344 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:27:45.855845 26344 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [CANDIDATE]: Term 3 election: Requested vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:27:45.856221 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "b77f8f52372241d483f457389fda0f93" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "1398472b92c842f285b8c7cc43cc1527"
I20240502 21:27:45.856428 24783 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:45.856880 24384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "b77f8f52372241d483f457389fda0f93" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:27:45.857079 24384 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:45.859148 24783 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77f8f52372241d483f457389fda0f93 in term 3.
I20240502 21:27:45.859724 24384 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77f8f52372241d483f457389fda0f93 in term 3.
I20240502 21:27:45.860170 26277 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, b77f8f52372241d483f457389fda0f93, e856847c3f494e6aa57f9cecf117ce4d; no voters:
I20240502 21:27:45.860576 31022 raft_consensus.cc:2781] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:45.861040 31022 raft_consensus.cc:686] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 3 LEADER]: Becoming Leader. State: Replica: b77f8f52372241d483f457389fda0f93, State: Running, Role: LEADER
I20240502 21:27:45.861472 31022 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:27:45.862879 26612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "b77f8f52372241d483f457389fda0f93" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "9b0952b24bd8400c9940dbd5e28417b4"
I20240502 21:27:45.863133 26612 raft_consensus.cc:3032] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:45.863233 26612 raft_consensus.cc:729] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9b0952b24bd8400c9940dbd5e28417b4, State: Running, Role: LEADER
I20240502 21:27:45.863543 26612 consensus_queue.cc:260] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:27:45.863909 26612 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:45.864873 24246 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 reported cstate change: term changed from 2 to 3, leader changed from 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) to b77f8f52372241d483f457389fda0f93 (127.16.122.5). New cstate: current_term: 3 leader_uuid: "b77f8f52372241d483f457389fda0f93" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:45.866581 26612 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77f8f52372241d483f457389fda0f93 in term 3.
W20240502 21:27:45.940644 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:46.063061 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:46.245671 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:46.253995 24783 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 21:27:46.254532 31022 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:46.367584 24384 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 21:27:46.368288 31022 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:46.373414 26612 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 3 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 21:27:46.374341 31022 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:27:46.378691 26344 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:27:46.381343 24783 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 21:27:46.381498 24384 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 21:27:46.382109 31460 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:46.383168 31463 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:27:46.385733 31460 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:46.387092 24783 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:46.387816 24384 raft_consensus.cc:2932] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } } }
I20240502 21:27:46.389180 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b6a4a74e0544e0e99e734a145c91d3b with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:46.390779 24246 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 12 to 14, VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New cstate: current_term: 3 leader_uuid: "b77f8f52372241d483f457389fda0f93" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:46.397125 26592 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6a4a74e0544e0e99e734a145c91d3b with delete_type TABLET_DATA_TOMBSTONED (TS 9b0952b24bd8400c9940dbd5e28417b4 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:58910
I20240502 21:27:46.397909 31470 tablet_replica.cc:330] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: stopping tablet replica
I20240502 21:27:46.398151 31470 raft_consensus.cc:2229] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:46.398427 31470 raft_consensus.cc:2258] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:46.399396 31470 ts_tablet_manager.cc:1875] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:46.403934 31470 ts_tablet_manager.cc:1888] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 21:27:46.404058 31470 log.cc:1196] T 2b6a4a74e0544e0e99e734a145c91d3b P 9b0952b24bd8400c9940dbd5e28417b4: Deleting WAL directory at /tmp/dist-test-task3Dzs1I/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714685122370222-16872-0/raft_consensus-itest-cluster/ts-6/wal/wals/2b6a4a74e0544e0e99e734a145c91d3b
I20240502 21:27:46.404807 24233 catalog_manager.cc:4830] TS 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): tablet 2b6a4a74e0544e0e99e734a145c91d3b (table rebalance_test_table_4 [id=a9c8cfe1b8fd490ea4bcba86677117a9]) successfully deleted
W20240502 21:27:46.481806 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:46.590788 24764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:46.615811 24632 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:46.625986 26722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:46.630232 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:27:46.631424 26325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:46.631606 26592 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:27:46.640213 24500 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:46.657066 24370 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:46.976532 26796 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3Dzs1I/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 21:27:15.115495 26802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:15.118693 26802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:15.143522 26802 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:18.676105 26802 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:27:21.339604 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:21.339946 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:21.340230 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:21.340401 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:21.340593 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:21.340703 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:21.340833 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:21.341038 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:21.341162 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:21.341372 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:21.341501 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:21.341619 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:21.341733 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:21.341863 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:21.342006 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fefd52238f245dfaf693d8dd980f834: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:21.342200 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:21.342314 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:21.360836 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server e856847c3f494e6aa57f9cecf117ce4d
I20240502 21:27:21.717497 26802 rebalancer_tool.cc:1074] tablet 0fc717a1b12e4380a4107e6deae2b837: 'e856847c3f494e6aa57f9cecf117ce4d' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:22.244074 26802 rebalancer_tool.cc:1074] tablet 7a35079a106c4cca94cf65ffa93aefbf: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:23.124220 26802 rebalancer_tool.cc:1074] tablet 82fb425a5430426f8030ba5497736b11: '1398472b92c842f285b8c7cc43cc1527' -> 'b77f8f52372241d483f457389fda0f93' move scheduled
I20240502 21:27:24.020023 26802 rebalancer_tool.cc:1074] tablet 616e911db4384c17905713adc595ea0a: '7f8bb40905da4891b461220b95c66832' -> '1769ee000eb24298a367a0de1209e6e0' move scheduled
I20240502 21:27:24.304085 26802 rebalancer_tool.cc:1084] tablet dc5c40e2bf0340958ecd3e7541209e95: 'e856847c3f494e6aa57f9cecf117ce4d' -> '9b0952b24bd8400c9940dbd5e28417b4' move ignored: Not found: Peer with uuid e856847c3f494e6aa57f9cecf117ce4d not found in consensus config
I20240502 21:27:24.304282 26802 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:27.021225 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:27.021387 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:27.021587 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:27.021708 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:27.021782 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:27.021950 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a90449876554e7393730fffdb194d49: could not find any suitable replica to move from server 1398472b92c842f285b8c7cc43cc1527 to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:27.022085 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
I20240502 21:27:27.662765 26802 rebalancer_tool.cc:1074] tablet 566929e603d54be4bf931413f0fa1d7d: 'e856847c3f494e6aa57f9cecf117ce4d' -> '' move scheduled
I20240502 21:27:28.061260 26802 rebalancer_tool.cc:1074] tablet 477d9df7d3a74b8584f3669be3256e00: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move scheduled
I20240502 21:27:28.270761 26802 rebalancer_tool.cc:1084] tablet 6c4aa176fd994cf0b55f304cf9d06fe0: '1398472b92c842f285b8c7cc43cc1527' -> '836775e329f148e5908a1afc54f06428' move ignored: Illegal state: tablet 6c4aa176fd994cf0b55f304cf9d06fe0: replica 836775e329f148e5908a1afc54f06428 is already present
I20240502 21:27:28.270897 26802 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:32.449347 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57c59a3b5480478d9f9953356d0daddc: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:32.449595 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 915a69d8d0034f63a71fc65a9234e61e: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 836775e329f148e5908a1afc54f06428
W20240502 21:27:32.450932 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c5ed106d16b346769829dc64c1ac81d3: could not find any suitable replica to move from server 7f8bb40905da4891b461220b95c66832 to server b77f8f52372241d483f457389fda0f93
W20240502 21:27:32.451097 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9c8cfe1b8fd490ea4bcba86677117a9: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:32.451192 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60beec8dbb7f48e1be85a9508b5b7fcd: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 9b0952b24bd8400c9940dbd5e28417b4
W20240502 21:27:32.451267 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7d3c1d066dcb423d87e5bd15cd684ec7: could not find any suitable replica to move from server e856847c3f494e6aa57f9cecf117ce4d to server 836775e329f148e5908a1afc54f06428
I20240502 21:27:32.837237 26802 rebalancer_tool.cc:1074] tablet 5c6e7eba855e48c0b0e18fa363ea01d4: '427345e9a65d417c84a773ac1925a18a' -> '836775e329f148e5908a1afc54f06428' move scheduled
I20240502 21:27:33.056007 26802 rebalancer_tool.cc:1074] tablet ac35b080037646b4b8b34c0eab2de4e6: '7f8bb40905da4891b461220b95c66832' -> '1398472b92c842f285b8c7cc43cc1527' move scheduled
I20240502 21:27:33.419371 26802 rebalancer_tool.cc:1084] tablet 6c4aa176fd994cf0b55f304cf9d06fe0: 'e856847c3f494e6aa57f9cecf117ce4d' -> '1769ee000eb24298a367a0de1209e6e0' move ignored: Illegal state: tablet 6c4aa176fd994cf0b55f304cf9d06fe0: replica 1769ee000eb24298a367a0de1209e6e0 is already present
I20240502 21:27:33.419505 26802 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:35.687497 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8931a08fc8114700bc668777de02e51c: could not find any suitable replica to move from server 427345e9a65d417c84a773ac1925a18a to server 1769ee000eb24298a367a0de1209e6e0
W20240502 21:27:35.687672 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 1398472b92c842f285b8c7cc43cc1527
I20240502 21:27:36.072232 26802 rebalancer_tool.cc:1084] tablet 8f912eb05c574368909e0b3a8edf058d: 'e856847c3f494e6aa57f9cecf117ce4d' -> 'b77f8f52372241d483f457389fda0f93' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:27:36.072409 26802 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:27:38.183714 26802 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63de16df705b4df0abcb6fcd022351ac: could not find any suitable replica to move from server 9b0952b24bd8400c9940dbd5e28417b4 to server 427345e9a65d417c84a773ac1925a18a
I20240502 21:27:38.299971 26802 rebalancer_tool.cc:1074] tablet 2b6a4a74e0544e0e99e734a145c91d3b: 'e856847c3f494e6aa57f9cecf117ce4d' -> '' move scheduled
I20240502 21:27:39.257277 26802 rebalancer_tool.cc:1136] tablet 7a35079a106c4cca94cf65ffa93aefbf: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: Incomplete: tablet 7a35079a106c4cca94cf65ffa93aefbf, TS 427345e9a65d417c84a773ac1925a18a -> TS 9b0952b24bd8400c9940dbd5e28417b4 move failed, target replica disappeared
I20240502 21:27:39.528578 26802 rebalancer_tool.cc:1136] tablet 0fc717a1b12e4380a4107e6deae2b837: 'e856847c3f494e6aa57f9cecf117ce4d' -> '836775e329f148e5908a1afc54f06428' move completed: Incomplete: tablet 0fc717a1b12e4380a4107e6deae2b837, TS e856847c3f494e6aa57f9cecf117ce4d -> TS 836775e329f148e5908a1afc54f06428 move failed, target replica disappeared
I20240502 21:27:39.800376 26802 rebalancer_tool.cc:1136] tablet 566929e603d54be4bf931413f0fa1d7d: 'e856847c3f494e6aa57f9cecf117ce4d' -> '' move completed: OK
I20240502 21:27:40.015173 26802 rebalancer_tool.cc:1136] tablet 477d9df7d3a74b8584f3669be3256e00: '427345e9a65d417c84a773ac1925a18a' -> '9b0952b24bd8400c9940dbd5e28417b4' move completed: OK
I20240502 21:27:40.387792 26802 rebalancer_tool.cc:1136] tablet 82fb425a5430426f8030ba5497736b11: '1398472b92c842f285b8c7cc43cc1527' -> 'b77f8f52372241d483f457389fda0f93' move completed: OK
I20240502 21:27:40.639549 26802 rebalancer_tool.cc:1136] tablet 5c6e7eba855e48c0b0e18fa363ea01d4: '427345e9a65d417c84a773ac1925a18a' -> '836775e329f148e5908a1afc54f06428' move completed: OK
I20240502 21:27:40.792939 26802 rebalancer_tool.cc:1136] tablet 616e911db4384c17905713adc595ea0a: '7f8bb40905da4891b461220b95c66832' -> '1769ee000eb24298a367a0de1209e6e0' move completed: OK
I20240502 21:27:40.919248 26802 rebalancer_tool.cc:1136] tablet ac35b080037646b4b8b34c0eab2de4e6: '7f8bb40905da4891b461220b95c66832' -> '1398472b92c842f285b8c7cc43cc1527' move completed: OK
I20240502 21:27:40.996076 26802 rebalancer_tool.cc:1136] tablet 2b6a4a74e0544e0e99e734a145c91d3b: 'e856847c3f494e6aa57f9cecf117ce4d' -> '' move completed: OK
I20240502 21:27:43.158841 26802 rebalancer_tool.cc:1074] tablet 2b6a4a74e0544e0e99e734a145c91d3b: '9b0952b24bd8400c9940dbd5e28417b4' -> 'e856847c3f494e6aa57f9cecf117ce4d' move scheduled
I20240502 21:27:43.290294 26802 rebalancer_tool.cc:1074] tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: '1769ee000eb24298a367a0de1209e6e0' -> '1398472b92c842f285b8c7cc43cc1527' move scheduled
I20240502 21:27:45.040918 26802 rebalancer_tool.cc:1136] tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: '1769ee000eb24298a367a0de1209e6e0' -> '1398472b92c842f285b8c7cc43cc1527' move completed: OK
I20240502 21:27:46.439129 26802 rebalancer_tool.cc:1136] tablet 2b6a4a74e0544e0e99e734a145c91d3b: '9b0952b24bd8400c9940dbd5e28417b4' -> 'e856847c3f494e6aa57f9cecf117ce4d' move completed: OK
Illegal state: tablet server 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): unacceptable health status UNAVAILABLE
W20240502 21:27:47.054294 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:47.130017 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:47.146323 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:47.746296 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:47.802642 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:47.825755 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.069836 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.127557 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.141727 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.168133 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.181156 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.197656 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:48.203759 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.366195 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:48.493458 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 21:27:48.566124 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 21:27:48.612736 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 21:27:48.686597 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:27:48.741328 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:27:48.805968 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:48.819880 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:27:48.895884 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:49.189977 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:49.265967 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:49.705730 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:49.733862 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:50.118199 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.179435 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:50.359221 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.420960 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.521073 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.557667 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.599629 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.629207 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.678391 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:50.690379 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:50.790067 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:51.024818 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:51.165088 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:51.719132 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:52.126941 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:52.275352 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:52.528874 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:52.627076 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:52.755872 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:52.832999 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:52.907835 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.040509 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.043360 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.215013 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:53.242746 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.288050 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.301143 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 21:27:53.302317 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:53.440780 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.487583 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:53.651825 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:27:53.710815 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:27:53.711719 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:53.768054 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:27:54.077148 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:27:54.103854 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:54.235047 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:27:54.381263 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:27:54.585861 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:54.817950 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.046092 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.118968 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:27:55.162860 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:55.244908 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.435297 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.451709 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.610155 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.654697 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:55.744366 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.818240 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:55.904242 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:56.222069 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:56.336649 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:56.628120 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:57.091288 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:57.141534 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:57.446755 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:57.689143 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:57.700202 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:27:57.793598 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:57.879370 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:57.975256 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:58.166543 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:58.310274 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:58.437016 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:58.591048 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:58.712173 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:27:58.713089 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:58.745977 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:27:58.747244 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:58.835969 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:27:58.974551 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:27:59.089324 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:59.314965 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:27:59.318048 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:27:59.374032 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:27:59.385972 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:27:59.519315 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:27:59.884385 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:27:59.923460 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:00.226022 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:00.288668 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:00.296499 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:00.320458 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:00.338325 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:28:00.810307 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:00.908151 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:01.099627 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:01.116887 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:01.212021 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:01.368269 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:01.395874 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:01.522754 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:01.524021 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:01.550849 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:01.968850 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:02.344797 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:02.432333 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:02.466084 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:02.619040 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:28:02.694864 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:02.775472 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:03.056305 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:03.271054 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:03.449402 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:03.547154 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:03.734340 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:03.735286 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:03.784765 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:03.852191 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:03.852938 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:03.927407 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:28:03.928354 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:03.951099 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:04.427439 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:04.458470 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:04.738606 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:04.867107 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:04.911301 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:05.012346 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:05.019688 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:05.028290 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:28:05.053876 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:05.124537 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:05.366528 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:05.806610 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:05.868252 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:06.086997 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:06.224514 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:06.307907 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:06.401615 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:06.439054 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:06.509028 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:06.839418 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:07.423543 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:28:07.461453 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:07.492842 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:07.554904 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:07.614907 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:07.706981 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:08.035200 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:08.188490 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:08.453037 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:08.492959 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:08.612519 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:08.747197 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:08.806339 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:08.834089 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:08.835166 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:08.976714 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:09.006985 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:28:09.182663 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:09.560566 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:09.620566 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:10.049376 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:28:10.140830 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:10.141858 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:10.182739 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:10.203300 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:10.212945 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:10.273320 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:10.372233 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:10.749631 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:10.892342 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:10.980852 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:11.026147 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:11.110306 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:11.271147 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:11.355439 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:11.554180 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:11.871219 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:12.355526 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:12.743641 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:28:12.788985 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:12.855873 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:12.874681 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:12.887362 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:12.911523 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:13.433739 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:13.465703 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:13.507684 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:13.538424 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:13.726060 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:13.827139 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:13.894593 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:28:13.895715 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:13.956553 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:14.172641 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:14.281036 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:28:14.566649 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:14.572377 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 21:28:15.145341 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:15.164422 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:15.165390 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:15.255569 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:28:15.432098 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:15.498481 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:15.589167 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:15.625370 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:15.654486 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:15.655428 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:15.813583 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:16.069664 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:16.122308 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:16.227329 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:16.241693 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:16.279923 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:16.669286 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:17.290863 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:17.577450 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:17.718214 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:17.785429 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:17.789544 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:28:17.950124 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:17.951650 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:17.967592 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:17.972147 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:18.073983 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:18.148816 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:18.314617 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:18.433220 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:18.649950 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:18.712252 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:18.757422 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:18.987505 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:18.988730 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:19.256585 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:19.358597 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:19.456816 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:28:19.768102 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:19.800124 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:20.197193 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:28:20.213604 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:20.215008 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:20.237974 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:20.247424 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:20.302289 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:20.596704 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:20.672616 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:20.689394 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:20.747216 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:20.748361 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:20.826397 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:20.999629 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:21.251176 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:21.336616 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:21.836652 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:21.936415 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:22.367084 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:22.450894 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:28:22.646761 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:22.743233 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:22.890408 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:22.984949 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:22.995013 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:22.998464 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:23.169986 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:23.229013 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:23.231142 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:23.562841 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:23.615134 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:23.744238 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:24.123334 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:24.124343 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:24.199187 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:24.584725 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:24.659408 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:28:24.711084 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:24.819597 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:24.870846 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:28:25.166066 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:25.387531 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.388163 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:25.389369 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.421242 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.424578 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.612007 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:25.613399 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.647143 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:25.780475 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.852363 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:25.854102 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:25.951871 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:26.095805 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:26.692660 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:26.841972 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:27.309159 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:27.389134 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:28:27.788383 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:27.921761 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:27.931030 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:27.951431 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:27.983103 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:28.008762 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:28.151816 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:28.177299 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:28.286062 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:28.339150 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:28.487757 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:28.904253 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:29.427474 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:29.428849 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:29.491454 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:29.783946 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:28:29.801703 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:28:29.887514 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:28:29.924384 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:30.020948 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:30.262826 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:30.277410 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.315582 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.411666 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.453125 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.587050 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:30.686393 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.817487 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.820536 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:30.835283 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:30.921773 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:31.191921 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:31.332749 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:31.705670 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:31.917311 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:32.274127 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:32.339104 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:28:32.518615 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:32.595690 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:32.837101 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:32.851471 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:32.960431 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:32.976845 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:33.025746 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:33.060591 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:33.129892 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:33.288774 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:33.455005 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:33.845752 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:34.227011 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:34.427914 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:34.428908 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:34.754155 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:34.829416 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:34.831640 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:28:34.832502 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:28:34.897521 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.233260 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:35.244482 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.328531 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:35.344521 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:28:35.378736 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.517144 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.630529 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.693918 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.718014 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.729743 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:35.803139 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:36.164301 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:36.222802 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:36.686666 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:36.897174 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:37.087261 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:37.191386 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:37.274921 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:28:37.308148 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:37.537336 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:37.593911 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:37.600241 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:37.693223 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:37.919154 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:38.023999 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:38.027225 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:38.028373 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:38.188508 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:38.445979 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:38.697675 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:39.250306 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:39.727059 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:39.732805 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:39.737619 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:39.779341 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:39.780256 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:39.871328 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:28:40.007277 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:28:40.089323 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:40.135478 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:40.146813 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:40.390991 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:40.394183 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:40.470645 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:40.572996 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:40.597698 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:40.711843 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:40.952010 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:41.217916 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:41.416817 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:41.652225 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:41.811838 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:41.940138 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:41.983546 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:42.029250 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:42.211555 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:42.374938 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:42.404852 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:28:42.459249 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:42.673395 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:42.729274 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:42.970697 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:43.038166 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:43.139394 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:43.168133 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:43.236903 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:43.427660 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:43.640116 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:44.181838 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:44.460721 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:44.614583 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:44.619136 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:44.889182 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:44.890105 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:28:44.927469 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:45.095377 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:28:45.140077 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:45.232457 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:45.376176 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:45.403293 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:45.404387 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:45.408969 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:45.427944 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:45.679138 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:45.694517 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:45.895606 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:45.970817 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:46.118611 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:46.676033 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:46.783666 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:47.131031 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:47.146817 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:47.182729 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:47.207181 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:47.322244 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:28:47.464514 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:47.589078 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:47.797009 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:47.826265 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:47.867812 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:47.944778 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:47.954638 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:47.978602 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:48.265512 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:48.425297 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:48.773067 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:48.868242 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:49.324016 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:49.345646 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:49.548194 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 21:28:49.826104 24237 maintenance_manager.cc:419] P 34199807c207492781e8dba15a3f67ab: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033375
I20240502 21:28:49.866878 24236 maintenance_manager.cc:643] P 34199807c207492781e8dba15a3f67ab: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.023s sys 0.015s Metrics: {"bytes_written":20181,"cfile_init":1,"dirs.queue_time_us":351,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1077,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1255,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":43,"thread_start_us":530,"threads_started":2,"wal-append.queue_time_us":462}
I20240502 21:28:49.867981 24237 maintenance_manager.cc:419] P 34199807c207492781e8dba15a3f67ab: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 12788 bytes on disk
I20240502 21:28:49.868860 24236 maintenance_manager.cc:643] P 34199807c207492781e8dba15a3f67ab: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240502 21:28:49.926427 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:49.927867 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:49.928838 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:28:49.943204 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:50.137686 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:50.171970 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:28:50.225502 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:50.332975 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:50.379487 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:50.496068 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:50.497164 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:50.695277 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:50.748104 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:50.749435 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:50.878108 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:50.921041 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:51.225477 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:51.319851 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:51.320672 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:51.844928 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:51.857080 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:51.889899 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:52.375877 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:52.414811 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:52.453953 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:52.476266 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:28:52.503432 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:52.533819 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:52.895381 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:52.902395 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:53.058840 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:53.094194 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:53.335500 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:53.403936 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:53.644075 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:53.902577 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:53.910485 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:54.244834 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:54.272627 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:54.302428 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:54.809496 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:54.869766 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:55.048032 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:28:55.048931 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:28:55.181208 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:28:55.186841 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:55.253984 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:55.419760 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:55.573283 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:55.607704 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:55.610286 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:55.734479 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:28:55.735529 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:56.106388 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:56.213847 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:56.471659 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:56.659241 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:56.720293 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:28:56.721406 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:56.748145 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:57.102239 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:28:57.367810 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:57.492424 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:28:57.524513 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:57.601226 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:28:57.778033 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:57.992394 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:58.077883 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:28:58.160221 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(7bf18ffb96d04d5e9ec8ee9563812ba0): perf score=0.033376
I20240502 21:28:58.160738 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(7bf18ffb96d04d5e9ec8ee9563812ba0): perf score=0.033372
W20240502 21:28:58.165704 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:58.170902 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:58.235692 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:28:58.290496 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(7bf18ffb96d04d5e9ec8ee9563812ba0) complete. Timing: real 0.129s user 0.070s sys 0.049s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":395,"dirs.queue_time_us":34763,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":4627,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1086,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":46564,"threads_started":3,"wal-append.queue_time_us":11797}
I20240502 21:28:58.292568 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(7bf18ffb96d04d5e9ec8ee9563812ba0): 4925 bytes on disk
I20240502 21:28:58.293648 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(7bf18ffb96d04d5e9ec8ee9563812ba0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 21:28:58.342612 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(7bf18ffb96d04d5e9ec8ee9563812ba0) complete. Timing: real 0.182s user 0.064s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":2596,"dirs.queue_time_us":459,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1081,"drs_written":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1026,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11679,"threads_started":3,"wal-append.queue_time_us":8821}
I20240502 21:28:58.345144 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(7bf18ffb96d04d5e9ec8ee9563812ba0): 4925 bytes on disk
I20240502 21:28:58.347406 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(7bf18ffb96d04d5e9ec8ee9563812ba0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240502 21:28:58.450204 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:58.512553 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:58.924798 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:28:59.093595 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:28:59.161756 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 21:28:59.298766 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(9d8a25580bc74004b0e31c34f841d2cc): perf score=0.033350
W20240502 21:28:59.329571 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 21:28:59.335925 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(9d8a25580bc74004b0e31c34f841d2cc) complete. Timing: real 0.037s user 0.027s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":407,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":1752,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":961,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":547,"threads_started":2,"wal-append.queue_time_us":368}
I20240502 21:28:59.337071 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(9d8a25580bc74004b0e31c34f841d2cc): 4840 bytes on disk
I20240502 21:28:59.337761 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(9d8a25580bc74004b0e31c34f841d2cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
W20240502 21:28:59.348472 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:28:59.412741 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(2e5a87719c2544a5b0cb591ed9f46df8): perf score=0.033385
I20240502 21:28:59.431900 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(7a35079a106c4cca94cf65ffa93aefbf): perf score=0.033397
I20240502 21:28:59.501324 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(2e5a87719c2544a5b0cb591ed9f46df8) complete. Timing: real 0.088s user 0.075s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":439,"dirs.queue_time_us":6400,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":6374,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":964,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10593,"threads_started":3,"wal-append.queue_time_us":3538}
I20240502 21:28:59.503610 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(2e5a87719c2544a5b0cb591ed9f46df8): 4586 bytes on disk
I20240502 21:28:59.505549 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(2e5a87719c2544a5b0cb591ed9f46df8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240502 21:28:59.547626 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:28:59.553705 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.121s user 0.071s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":437,"dirs.queue_time_us":14425,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":935,"drs_written":1,"lbm_read_time_us":653,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1051,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":33913,"threads_started":3,"wal-append.queue_time_us":19447}
I20240502 21:28:59.555611 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf): 4925 bytes on disk
I20240502 21:28:59.556617 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 21:29:00.062897 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:29:00.064078 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:00.074064 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:29:00.150580 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:29:00.355612 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(855b247f654d41c2bbb13abb25d35c1c): perf score=0.033334
I20240502 21:29:00.386193 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(855b247f654d41c2bbb13abb25d35c1c) complete. Timing: real 0.030s user 0.027s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":352,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1792,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":610,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":469,"threads_started":2,"wal-append.queue_time_us":339}
I20240502 21:29:00.387408 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(855b247f654d41c2bbb13abb25d35c1c): 4587 bytes on disk
I20240502 21:29:00.388025 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(855b247f654d41c2bbb13abb25d35c1c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240502 21:29:00.391460 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 21:29:00.497084 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:29:00.510706 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(6c4aa176fd994cf0b55f304cf9d06fe0): perf score=0.033375
I20240502 21:29:00.545027 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.034s user 0.019s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1791,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":674,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":487,"threads_started":2,"wal-append.queue_time_us":362}
I20240502 21:29:00.546296 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0): 4925 bytes on disk
I20240502 21:29:00.546886 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 21:29:00.561578 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(7b2661e0b5fe4a0b94c150b14b8fd9ec): perf score=0.033386
W20240502 21:29:00.567279 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:29:00.593340 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(855b247f654d41c2bbb13abb25d35c1c): perf score=0.033396
I20240502 21:29:00.620748 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(7b2661e0b5fe4a0b94c150b14b8fd9ec) complete. Timing: real 0.059s user 0.049s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":404,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1598,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":521,"threads_started":2,"wal-append.queue_time_us":387}
I20240502 21:29:00.622701 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(7b2661e0b5fe4a0b94c150b14b8fd9ec): 4841 bytes on disk
I20240502 21:29:00.627504 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(7b2661e0b5fe4a0b94c150b14b8fd9ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240502 21:29:00.634900 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:00.650095 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(855b247f654d41c2bbb13abb25d35c1c) complete. Timing: real 0.056s user 0.036s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5226,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":7597,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6543,"threads_started":2,"wal-append.queue_time_us":368}
I20240502 21:29:00.651363 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(6c4aa176fd994cf0b55f304cf9d06fe0): perf score=0.033410
W20240502 21:29:00.675611 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:29:00.691007 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.039s user 0.031s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1735,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":683,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":281,"threads_started":1,"wal-append.queue_time_us":401}
I20240502 21:29:00.692250 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0): 4925 bytes on disk
I20240502 21:29:00.692818 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 21:29:00.693672 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(855b247f654d41c2bbb13abb25d35c1c): 4587 bytes on disk
I20240502 21:29:00.694237 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(855b247f654d41c2bbb13abb25d35c1c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240502 21:29:00.915599 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:00.916575 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:29:01.031276 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:29:01.082767 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:01.359650 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:29:01.381172 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(5c6e7eba855e48c0b0e18fa363ea01d4): perf score=0.033351
I20240502 21:29:01.392452 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(adde18faca794600bb4aa24f0cf744e6): perf score=0.033350
I20240502 21:29:01.416728 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(5c6e7eba855e48c0b0e18fa363ea01d4) complete. Timing: real 0.035s user 0.021s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":366,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1780,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":527,"threads_started":2,"wal-append.queue_time_us":369}
I20240502 21:29:01.418412 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(5c6e7eba855e48c0b0e18fa363ea01d4): 4840 bytes on disk
I20240502 21:29:01.419138 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(5c6e7eba855e48c0b0e18fa363ea01d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
W20240502 21:29:01.423902 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 21:29:01.442413 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.050s user 0.042s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1448,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":921,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":495,"threads_started":2,"wal-append.queue_time_us":367}
I20240502 21:29:01.443455 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6): 4586 bytes on disk
I20240502 21:29:01.444139 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 21:29:01.447926 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(adde18faca794600bb4aa24f0cf744e6): perf score=0.033363
I20240502 21:29:01.479305 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.031s user 0.024s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":544,"threads_started":2,"wal-append.queue_time_us":440}
I20240502 21:29:01.480710 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6): 4586 bytes on disk
I20240502 21:29:01.481316 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240502 21:29:01.485752 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:01.550704 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(8f912eb05c574368909e0b3a8edf058d): perf score=0.033396
I20240502 21:29:01.585033 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(8f912eb05c574368909e0b3a8edf058d) complete. Timing: real 0.034s user 0.027s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":441,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1635,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":594,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":574,"threads_started":2,"wal-append.queue_time_us":412}
I20240502 21:29:01.586053 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(5c6e7eba855e48c0b0e18fa363ea01d4): perf score=0.033402
I20240502 21:29:01.653465 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(5c6e7eba855e48c0b0e18fa363ea01d4) complete. Timing: real 0.067s user 0.053s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1618,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":251,"threads_started":1,"wal-append.queue_time_us":363}
I20240502 21:29:01.654542 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(8f912eb05c574368909e0b3a8edf058d): 4925 bytes on disk
I20240502 21:29:01.655138 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(8f912eb05c574368909e0b3a8edf058d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 21:29:01.655941 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(5c6e7eba855e48c0b0e18fa363ea01d4): 4840 bytes on disk
I20240502 21:29:01.656380 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(5c6e7eba855e48c0b0e18fa363ea01d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 21:29:01.858330 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:01.859519 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:01.954077 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:02.053750 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:02.539672 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:29:02.567005 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:02.731690 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:02.846365 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:02.990621 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:03.058137 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:03.088912 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:03.094770 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:03.225028 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:03.297758 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:03.397105 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:03.590502 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:03.615989 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 21:29:03.663321 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(dc5c40e2bf0340958ecd3e7541209e95): perf score=0.033365
I20240502 21:29:03.677947 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(2b6a4a74e0544e0e99e734a145c91d3b): perf score=0.033365
I20240502 21:29:03.701623 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(dc5c40e2bf0340958ecd3e7541209e95) complete. Timing: real 0.038s user 0.027s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":412,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1631,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2463,"threads_started":2,"wal-append.queue_time_us":2259}
I20240502 21:29:03.702939 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(dc5c40e2bf0340958ecd3e7541209e95): 4925 bytes on disk
I20240502 21:29:03.703550 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(dc5c40e2bf0340958ecd3e7541209e95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 21:29:03.718694 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(2b6a4a74e0544e0e99e734a145c91d3b) complete. Timing: real 0.040s user 0.030s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":411,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1354,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":799,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":536,"threads_started":2,"wal-append.queue_time_us":361}
I20240502 21:29:03.719820 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(2b6a4a74e0544e0e99e734a145c91d3b): 4584 bytes on disk
I20240502 21:29:03.720352 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(2b6a4a74e0544e0e99e734a145c91d3b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240502 21:29:04.055809 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:04.058871 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:04.454119 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:04.583838 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:04.630268 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:29:04.705905 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(616e911db4384c17905713adc595ea0a): perf score=0.033369
I20240502 21:29:04.725260 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(d0ba1af1d3624d6387bf892873f3f67e): perf score=0.033380
I20240502 21:29:04.743664 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(616e911db4384c17905713adc595ea0a): perf score=0.033379
I20240502 21:29:04.749967 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.044s user 0.025s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":506,"threads_started":2,"wal-append.queue_time_us":404}
I20240502 21:29:04.751155 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a): 4841 bytes on disk
I20240502 21:29:04.751716 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 21:29:04.752573 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0): perf score=0.033380
I20240502 21:29:04.788651 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.045s user 0.033s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1396,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1330,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":932,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1588,"threads_started":2,"wal-append.queue_time_us":400}
I20240502 21:29:04.790220 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a): 4841 bytes on disk
I20240502 21:29:04.790838 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 21:29:04.807665 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(d0ba1af1d3624d6387bf892873f3f67e) complete. Timing: real 0.082s user 0.073s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4330,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3124,"threads_started":2,"wal-append.queue_time_us":1458}
I20240502 21:29:04.809226 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(d0ba1af1d3624d6387bf892873f3f67e): 4925 bytes on disk
I20240502 21:29:04.809907 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(d0ba1af1d3624d6387bf892873f3f67e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 21:29:04.821820 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0) complete. Timing: real 0.069s user 0.050s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1581,"drs_written":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4,"lbm_write_time_us":713,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":232,"threads_started":1,"wal-append.queue_time_us":381}
I20240502 21:29:04.823127 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0): 4586 bytes on disk
I20240502 21:29:04.823694 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
W20240502 21:29:04.875277 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:29:05.101516 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:05.102645 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:05.434238 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:29:05.435118 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:05.534029 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:05.613914 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:05.635643 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:05.685335 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:29:05.711161 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(53c1992a21224bac883da474f143c1d7): perf score=0.033365
W20240502 21:29:05.715008 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:05.739439 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:29:05.745584 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(53c1992a21224bac883da474f143c1d7) complete. Timing: real 0.034s user 0.027s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":421,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":1589,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":752,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":536,"threads_started":2,"wal-append.queue_time_us":425}
I20240502 21:29:05.746775 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(53c1992a21224bac883da474f143c1d7): 4925 bytes on disk
I20240502 21:29:05.747400 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(53c1992a21224bac883da474f143c1d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 21:29:05.795173 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(5dd91818e74e42acb276a3864753094d): perf score=0.033389
I20240502 21:29:05.814848 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(5dd91818e74e42acb276a3864753094d): perf score=0.033385
I20240502 21:29:05.828402 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(049bec57d2474c328d2e26e16a4d33a9): perf score=0.033394
I20240502 21:29:05.850133 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.055s user 0.035s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":347,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":3324,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2501,"threads_started":2,"wal-append.queue_time_us":1522}
I20240502 21:29:05.852031 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d): 4840 bytes on disk
I20240502 21:29:05.854249 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4}
I20240502 21:29:05.873829 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.059s user 0.046s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11723,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":3008,"drs_written":1,"lbm_read_time_us":216,"lbm_reads_lt_1ms":4,"lbm_write_time_us":942,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":11933,"threads_started":2,"wal-append.queue_time_us":408}
I20240502 21:29:05.875504 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d): 4840 bytes on disk
I20240502 21:29:05.876191 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 21:29:05.882292 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(049bec57d2474c328d2e26e16a4d33a9) complete. Timing: real 0.054s user 0.026s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2425,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":2317,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":818,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2493,"threads_started":2,"wal-append.queue_time_us":283}
I20240502 21:29:05.883448 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(049bec57d2474c328d2e26e16a4d33a9): 4586 bytes on disk
I20240502 21:29:05.884073 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(049bec57d2474c328d2e26e16a4d33a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240502 21:29:05.975078 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:06.197638 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:06.198258 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:06.395709 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:06.575121 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:06.757241 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(2ee3718b120e4fad9ae03e47d3f1fffa): perf score=0.033337
W20240502 21:29:06.769645 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:29:06.818253 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(2ee3718b120e4fad9ae03e47d3f1fffa) complete. Timing: real 0.061s user 0.029s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1515,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":611,"threads_started":2,"wal-append.queue_time_us":449}
I20240502 21:29:06.819315 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(2ee3718b120e4fad9ae03e47d3f1fffa): 4923 bytes on disk
I20240502 21:29:06.819959 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(2ee3718b120e4fad9ae03e47d3f1fffa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 21:29:06.859817 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(477d9df7d3a74b8584f3669be3256e00): perf score=0.033377
I20240502 21:29:06.881371 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(477d9df7d3a74b8584f3669be3256e00): perf score=0.033376
I20240502 21:29:06.888433 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(82fb425a5430426f8030ba5497736b11): perf score=0.033371
I20240502 21:29:06.911991 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.052s user 0.042s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":889,"threads_started":2,"wal-append.queue_time_us":778}
I20240502 21:29:06.913672 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(2ee3718b120e4fad9ae03e47d3f1fffa): perf score=0.033387
I20240502 21:29:06.931896 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.050s user 0.027s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":723,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1859,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3897,"threads_started":2,"wal-append.queue_time_us":3474}
I20240502 21:29:06.933887 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00): 4586 bytes on disk
I20240502 21:29:06.936558 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 21:29:06.943686 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(82fb425a5430426f8030ba5497736b11) complete. Timing: real 0.055s user 0.028s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9312,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":4623,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9675,"threads_started":2,"wal-append.queue_time_us":355}
I20240502 21:29:06.944804 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(82fb425a5430426f8030ba5497736b11): 4840 bytes on disk
I20240502 21:29:06.945284 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(82fb425a5430426f8030ba5497736b11) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 21:29:06.955842 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(2ee3718b120e4fad9ae03e47d3f1fffa) complete. Timing: real 0.042s user 0.035s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":262,"threads_started":1,"wal-append.queue_time_us":378}
I20240502 21:29:06.956959 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(2ee3718b120e4fad9ae03e47d3f1fffa): 4923 bytes on disk
I20240502 21:29:06.957563 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(2ee3718b120e4fad9ae03e47d3f1fffa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 21:29:06.958645 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00): 4586 bytes on disk
I20240502 21:29:06.959157 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240502 21:29:07.078166 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:07.306948 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:07.388335 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:07.487061 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:29:07.650955 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:07.671286 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:07.706836 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:08.039542 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:08.040560 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:08.134753 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:08.255198 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:08.295351 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:08.371400 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:08.492691 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:08.497439 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:08.573997 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:08.701710 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(4cf38bbaaaa340f89d766a843c6bcaf9): perf score=0.033340
I20240502 21:29:08.749428 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(4cf38bbaaaa340f89d766a843c6bcaf9) complete. Timing: real 0.047s user 0.028s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":378,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1566,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":648,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":450,"threads_started":2,"wal-append.queue_time_us":316}
I20240502 21:29:08.750914 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(4cf38bbaaaa340f89d766a843c6bcaf9): 4925 bytes on disk
I20240502 21:29:08.751470 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(4cf38bbaaaa340f89d766a843c6bcaf9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 21:29:08.927713 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 21:29:08.945168 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(ff7e019fa82d4648afaf130d3f6343d6): perf score=0.033397
I20240502 21:29:08.967118 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(4cf38bbaaaa340f89d766a843c6bcaf9): perf score=0.033402
I20240502 21:29:08.986129 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(ff7e019fa82d4648afaf130d3f6343d6) complete. Timing: real 0.041s user 0.031s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":681,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":2396,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":806,"threads_started":2,"wal-append.queue_time_us":446}
I20240502 21:29:08.987696 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(ff7e019fa82d4648afaf130d3f6343d6): 4585 bytes on disk
I20240502 21:29:08.988348 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(ff7e019fa82d4648afaf130d3f6343d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 21:29:09.012584 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(4cf38bbaaaa340f89d766a843c6bcaf9) complete. Timing: real 0.045s user 0.029s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1344,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":797,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":504,"threads_started":2,"wal-append.queue_time_us":409}
I20240502 21:29:09.013639 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(4cf38bbaaaa340f89d766a843c6bcaf9): 4925 bytes on disk
I20240502 21:29:09.014276 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(4cf38bbaaaa340f89d766a843c6bcaf9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240502 21:29:09.119402 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:09.412968 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:09.580277 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:09.584573 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:09.849324 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:29:10.146752 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:10.147873 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:10.171245 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:10.348048 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:10.464660 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:10.592275 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:29:10.615449 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:10.659559 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:10.884876 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:29:10.996591 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(e847a796029d4edea79694f5402a291a): perf score=0.033356
I20240502 21:29:11.023483 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(98b5668e0d504db5a7874f041c14dbfa): perf score=0.033363
I20240502 21:29:11.033142 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(e847a796029d4edea79694f5402a291a) complete. Timing: real 0.036s user 0.017s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":2073,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":999,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":512,"threads_started":2,"wal-append.queue_time_us":351}
I20240502 21:29:11.034267 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(e847a796029d4edea79694f5402a291a): 4925 bytes on disk
I20240502 21:29:11.034904 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(e847a796029d4edea79694f5402a291a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 21:29:11.056419 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.033s user 0.026s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":322,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":432,"threads_started":2,"wal-append.queue_time_us":315}
I20240502 21:29:11.057579 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa): 4840 bytes on disk
I20240502 21:29:11.058153 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240502 21:29:11.071856 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:29:11.084466 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(9963dcea478749e0a6e076fac6537fc4): perf score=0.033376
W20240502 21:29:11.111714 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:29:11.115518 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(9963dcea478749e0a6e076fac6537fc4) complete. Timing: real 0.031s user 0.022s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":360,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1507,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":677,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":5504,"thread_start_us":438,"threads_started":2,"wal-append.queue_time_us":325}
I20240502 21:29:11.116708 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(9963dcea478749e0a6e076fac6537fc4): 4586 bytes on disk
I20240502 21:29:11.117255 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(9963dcea478749e0a6e076fac6537fc4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240502 21:29:11.176807 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:11.435894 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:11.538007 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:11.781098 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:12.005087 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:12.058085 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:12.324734 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:12.463541 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:29:12.571039 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:12.827742 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:12.906415 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:12.913331 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:12.919905 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:13.109282 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:13.240360 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:13.244415 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:13.299068 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:13.450965 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:13.567600 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:13.602504 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:13.846454 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:14.131229 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:14.348361 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:14.389861 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:14.413082 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:14.819598 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:15.000161 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:29:15.296023 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:15.301318 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:15.302242 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:15.391602 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:15.464643 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:15.688804 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:15.696162 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:15.766270 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:29:15.899454 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:16.050427 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:16.052598 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:16.118045 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:16.517244 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:16.551100 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:16.660312 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:16.733201 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:16.788645 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:17.005112 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:17.412001 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:17.643126 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:29:17.673981 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:17.768211 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:17.835263 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:17.940522 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:17.948063 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:18.330972 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:18.357550 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:18.401376 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:18.439488 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:18.458892 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:18.728224 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:18.895915 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:18.975451 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:18.976379 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:19.120642 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:19.218909 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:19.233129 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:19.612617 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:20.098562 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:20.198776 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:20.384625 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:20.431893 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:20.476490 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:20.477346 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:20.484665 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:29:20.547099 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:20.758164 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:20.887823 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:20.911406 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:20.938431 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 21:29:21.038867 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(e847a796029d4edea79694f5402a291a): perf score=0.033379
I20240502 21:29:21.098134 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(e847a796029d4edea79694f5402a291a) complete. Timing: real 0.059s user 0.035s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":345,"dirs.queue_time_us":447,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1085,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1198,"threads_started":3,"wal-append.queue_time_us":778}
I20240502 21:29:21.099843 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(e847a796029d4edea79694f5402a291a): 4925 bytes on disk
I20240502 21:29:21.100545 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(e847a796029d4edea79694f5402a291a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240502 21:29:21.357842 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:21.465687 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 21:29:21.603232 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(2e5a87719c2544a5b0cb591ed9f46df8): perf score=0.033336
I20240502 21:29:21.636860 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(2e5a87719c2544a5b0cb591ed9f46df8) complete. Timing: real 0.033s user 0.022s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1826,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":450,"threads_started":2,"wal-append.queue_time_us":320}
I20240502 21:29:21.638110 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(2e5a87719c2544a5b0cb591ed9f46df8): 4586 bytes on disk
I20240502 21:29:21.638792 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(2e5a87719c2544a5b0cb591ed9f46df8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 21:29:21.640417 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:21.679389 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:29:21.746506 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:21.776983 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:29:21.946908 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(98b5668e0d504db5a7874f041c14dbfa): perf score=0.033349
W20240502 21:29:21.981123 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:29:22.028527 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.081s user 0.070s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":424,"dirs.queue_time_us":482,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":962,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":907,"threads_started":3,"wal-append.queue_time_us":380}
I20240502 21:29:22.030032 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa): 4840 bytes on disk
I20240502 21:29:22.030819 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 21:29:22.142431 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(ff7e019fa82d4648afaf130d3f6343d6): perf score=0.033348
I20240502 21:29:22.172905 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(0fc717a1b12e4380a4107e6deae2b837): perf score=0.033363
I20240502 21:29:22.210866 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(ff7e019fa82d4648afaf130d3f6343d6) complete. Timing: real 0.068s user 0.032s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18213,"threads_started":2,"wal-append.queue_time_us":18064}
I20240502 21:29:22.214788 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(ff7e019fa82d4648afaf130d3f6343d6): 4585 bytes on disk
I20240502 21:29:22.215749 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(ff7e019fa82d4648afaf130d3f6343d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240502 21:29:22.268150 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:22.275010 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(0fc717a1b12e4380a4107e6deae2b837) complete. Timing: real 0.101s user 0.069s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":416,"dirs.queue_time_us":6530,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":5295,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":949,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7151,"threads_started":3,"wal-append.queue_time_us":409}
I20240502 21:29:22.276897 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(0fc717a1b12e4380a4107e6deae2b837): 4586 bytes on disk
I20240502 21:29:22.277902 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(0fc717a1b12e4380a4107e6deae2b837) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 21:29:22.467959 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(7b2661e0b5fe4a0b94c150b14b8fd9ec): perf score=0.033375
I20240502 21:29:22.545380 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(7b2661e0b5fe4a0b94c150b14b8fd9ec) complete. Timing: real 0.077s user 0.048s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":781,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8006,"threads_started":1,"wal-append.queue_time_us":7845}
I20240502 21:29:22.546988 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(7b2661e0b5fe4a0b94c150b14b8fd9ec): 4841 bytes on disk
I20240502 21:29:22.547705 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(7b2661e0b5fe4a0b94c150b14b8fd9ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240502 21:29:22.714319 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:22.781085 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(566929e603d54be4bf931413f0fa1d7d): perf score=0.033382
W20240502 21:29:22.796581 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 21:29:22.834033 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(566929e603d54be4bf931413f0fa1d7d) complete. Timing: real 0.053s user 0.032s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":335,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1523,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":681,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":436,"threads_started":2,"wal-append.queue_time_us":296}
I20240502 21:29:22.834985 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(566929e603d54be4bf931413f0fa1d7d): 4840 bytes on disk
I20240502 21:29:22.835474 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(566929e603d54be4bf931413f0fa1d7d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240502 21:29:22.896234 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:22.932653 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:22.948688 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:22.978334 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:23.089377 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:23.300536 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:29:23.337690 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(dc5c40e2bf0340958ecd3e7541209e95): perf score=0.033341
W20240502 21:29:23.347919 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:29:23.370658 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(dc5c40e2bf0340958ecd3e7541209e95) complete. Timing: real 0.033s user 0.028s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":404,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1690,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":522,"threads_started":2,"wal-append.queue_time_us":368}
I20240502 21:29:23.371631 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(dc5c40e2bf0340958ecd3e7541209e95): 4925 bytes on disk
I20240502 21:29:23.372197 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(dc5c40e2bf0340958ecd3e7541209e95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240502 21:29:23.439256 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:23.449235 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:29:23.538082 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(855b247f654d41c2bbb13abb25d35c1c): perf score=0.033373
I20240502 21:29:23.580546 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(855b247f654d41c2bbb13abb25d35c1c) complete. Timing: real 0.042s user 0.036s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":386,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":2285,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":459,"threads_started":2,"wal-append.queue_time_us":301}
I20240502 21:29:23.581715 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(855b247f654d41c2bbb13abb25d35c1c): 4587 bytes on disk
I20240502 21:29:23.582281 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(855b247f654d41c2bbb13abb25d35c1c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 21:29:23.623708 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(adde18faca794600bb4aa24f0cf744e6): perf score=0.033349
I20240502 21:29:23.655665 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.032s user 0.022s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1769,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":222,"threads_started":1,"wal-append.queue_time_us":330}
I20240502 21:29:23.656672 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6): 4586 bytes on disk
I20240502 21:29:23.657292 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 21:29:23.660467 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:23.834460 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(ac35b080037646b4b8b34c0eab2de4e6): perf score=0.033380
I20240502 21:29:23.865777 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(ac35b080037646b4b8b34c0eab2de4e6) complete. Timing: real 0.031s user 0.017s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1458,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":421,"threads_started":1,"wal-append.queue_time_us":523}
I20240502 21:29:23.867055 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(ac35b080037646b4b8b34c0eab2de4e6): 4840 bytes on disk
I20240502 21:29:23.867730 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(ac35b080037646b4b8b34c0eab2de4e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
W20240502 21:29:23.963814 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:24.183998 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:24.236404 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:24.238133 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:24.344543 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:29:24.555958 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(82fb425a5430426f8030ba5497736b11): perf score=0.033370
I20240502 21:29:24.594038 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(82fb425a5430426f8030ba5497736b11) complete. Timing: real 0.038s user 0.015s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":546,"threads_started":2,"wal-append.queue_time_us":397}
I20240502 21:29:24.595160 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(82fb425a5430426f8030ba5497736b11): 4840 bytes on disk
I20240502 21:29:24.595722 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(82fb425a5430426f8030ba5497736b11) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240502 21:29:24.653162 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:24.735893 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:25.178050 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:25.250958 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:25.334208 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:29:25.688517 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:29:25.689775 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:25.756510 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:25.762362 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:25.800599 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:25.868839 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:26.038319 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:26.054368 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:29:26.112708 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:26.209040 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:26.442032 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:26.625957 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:26.650547 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:26.824558 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:27.006250 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:27.170116 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(616e911db4384c17905713adc595ea0a): perf score=0.033336
I20240502 21:29:27.207160 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.037s user 0.032s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":437,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1765,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":806,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":547,"threads_started":2,"wal-append.queue_time_us":363}
I20240502 21:29:27.208390 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a): 4841 bytes on disk
I20240502 21:29:27.209095 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 21:29:27.495431 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:27.538583 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:27.548310 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:27.904836 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:29:28.015059 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:28.043478 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 21:29:28.146665 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:28.149258 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:28.151402 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:28.339203 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:28.605126 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:28.610461 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:29:28.610806 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(53c1992a21224bac883da474f143c1d7): perf score=0.033367
W20240502 21:29:28.651080 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:28.653374 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 21:29:28.695673 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(53c1992a21224bac883da474f143c1d7) complete. Timing: real 0.085s user 0.065s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2813,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":8533,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1031,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3233,"threads_started":2,"wal-append.queue_time_us":402}
I20240502 21:29:28.697010 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(53c1992a21224bac883da474f143c1d7): 4925 bytes on disk
I20240502 21:29:28.697623 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(53c1992a21224bac883da474f143c1d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 21:29:28.992844 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:29.059077 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:29.211076 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:29:29.388722 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(049bec57d2474c328d2e26e16a4d33a9): perf score=0.033396
I20240502 21:29:29.420259 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(049bec57d2474c328d2e26e16a4d33a9) complete. Timing: real 0.031s user 0.015s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":439,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1622,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":468,"threads_started":2,"wal-append.queue_time_us":331}
I20240502 21:29:29.421344 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(049bec57d2474c328d2e26e16a4d33a9): 4586 bytes on disk
I20240502 21:29:29.421885 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(049bec57d2474c328d2e26e16a4d33a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240502 21:29:29.514328 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:29.673697 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(477d9df7d3a74b8584f3669be3256e00): perf score=0.033363
I20240502 21:29:29.702490 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(d0ba1af1d3624d6387bf892873f3f67e): perf score=0.033359
I20240502 21:29:29.717866 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(53c1992a21224bac883da474f143c1d7): perf score=0.033353
I20240502 21:29:29.728534 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.055s user 0.045s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":75,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":1488,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":730,"threads_started":1,"wal-append.queue_time_us":900}
I20240502 21:29:29.729733 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00): 4586 bytes on disk
I20240502 21:29:29.730490 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
I20240502 21:29:29.731557 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(5dd91818e74e42acb276a3864753094d): perf score=0.033379
I20240502 21:29:29.769328 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.038s user 0.033s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1377,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":294,"threads_started":1,"wal-append.queue_time_us":420}
I20240502 21:29:29.771026 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(566929e603d54be4bf931413f0fa1d7d): perf score=0.033369
I20240502 21:29:29.775159 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(d0ba1af1d3624d6387bf892873f3f67e) complete. Timing: real 0.072s user 0.043s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":18904,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1578,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1097,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":23681,"threads_started":2,"wal-append.queue_time_us":4479}
I20240502 21:29:29.776845 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(d0ba1af1d3624d6387bf892873f3f67e): 4925 bytes on disk
I20240502 21:29:29.777649 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(d0ba1af1d3624d6387bf892873f3f67e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
I20240502 21:29:29.782699 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(53c1992a21224bac883da474f143c1d7) complete. Timing: real 0.064s user 0.056s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1338,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1348,"threads_started":2,"wal-append.queue_time_us":729}
I20240502 21:29:29.784176 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(53c1992a21224bac883da474f143c1d7): 4925 bytes on disk
I20240502 21:29:29.784783 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(53c1992a21224bac883da474f143c1d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240502 21:29:29.803284 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:29:29.816689 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(566929e603d54be4bf931413f0fa1d7d) complete. Timing: real 0.045s user 0.031s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":250,"threads_started":1,"wal-append.queue_time_us":386}
I20240502 21:29:29.817992 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(566929e603d54be4bf931413f0fa1d7d): 4840 bytes on disk
I20240502 21:29:29.818709 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(566929e603d54be4bf931413f0fa1d7d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 21:29:29.819555 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d): 4840 bytes on disk
I20240502 21:29:29.820029 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 21:29:29.820837 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(d0ba1af1d3624d6387bf892873f3f67e): perf score=0.033375
W20240502 21:29:29.823637 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 21:29:29.853348 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(d0ba1af1d3624d6387bf892873f3f67e) complete. Timing: real 0.032s user 0.029s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":1572,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":306,"threads_started":1,"wal-append.queue_time_us":477}
I20240502 21:29:29.854517 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(d0ba1af1d3624d6387bf892873f3f67e): 4925 bytes on disk
I20240502 21:29:29.855059 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(d0ba1af1d3624d6387bf892873f3f67e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 21:29:29.855866 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(8f912eb05c574368909e0b3a8edf058d): perf score=0.033360
I20240502 21:29:29.901134 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(8f912eb05c574368909e0b3a8edf058d) complete. Timing: real 0.045s user 0.039s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1595,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":680,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":280,"threads_started":1,"wal-append.queue_time_us":386}
I20240502 21:29:29.902192 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(8f912eb05c574368909e0b3a8edf058d): 4925 bytes on disk
I20240502 21:29:29.902734 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(8f912eb05c574368909e0b3a8edf058d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240502 21:29:30.081100 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:30.251997 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:29:30.280748 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(0fc717a1b12e4380a4107e6deae2b837): perf score=0.033400
I20240502 21:29:30.311866 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(0fc717a1b12e4380a4107e6deae2b837) complete. Timing: real 0.031s user 0.025s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":425,"dirs.run_cpu_time_us":344,"dirs.run_wall_time_us":1652,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":564,"threads_started":2,"wal-append.queue_time_us":394}
I20240502 21:29:30.313113 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(0fc717a1b12e4380a4107e6deae2b837): 4586 bytes on disk
I20240502 21:29:30.313699 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(0fc717a1b12e4380a4107e6deae2b837) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240502 21:29:30.491148 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:30.536717 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:29:30.745609 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:30.788918 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:30.791599 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:30.847810 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:31.032038 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:31.052119 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:31.078071 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:29:31.079088 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:31.311359 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 21:29:31.318039 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(2b6a4a74e0544e0e99e734a145c91d3b): perf score=0.033374
I20240502 21:29:31.359134 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(2b6a4a74e0544e0e99e734a145c91d3b) complete. Timing: real 0.041s user 0.029s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":386,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1599,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":739,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6733,"threads_started":2,"wal-append.queue_time_us":6338}
I20240502 21:29:31.360510 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(2b6a4a74e0544e0e99e734a145c91d3b): 4584 bytes on disk
I20240502 21:29:31.361196 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(2b6a4a74e0544e0e99e734a145c91d3b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240502 21:29:31.547705 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:31.571208 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:31.682623 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:29:32.044044 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(e847a796029d4edea79694f5402a291a): perf score=0.033353
W20240502 21:29:32.051836 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:32.102784 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(e847a796029d4edea79694f5402a291a) complete. Timing: real 0.058s user 0.042s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":425,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1564,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":668,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":537,"threads_started":2,"wal-append.queue_time_us":332}
I20240502 21:29:32.103904 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(e847a796029d4edea79694f5402a291a): 4925 bytes on disk
I20240502 21:29:32.104449 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(e847a796029d4edea79694f5402a291a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 21:29:32.114946 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(bdf23b7a397441b5947ce6babe3e8f62): perf score=0.033361
W20240502 21:29:32.152412 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 21:29:32.170063 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(bdf23b7a397441b5947ce6babe3e8f62) complete. Timing: real 0.055s user 0.038s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":407,"dirs.run_cpu_time_us":307,"dirs.run_wall_time_us":2847,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":810,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1497,"threads_started":2,"wal-append.queue_time_us":1348}
I20240502 21:29:32.171413 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(bdf23b7a397441b5947ce6babe3e8f62): 4840 bytes on disk
I20240502 21:29:32.171963 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(bdf23b7a397441b5947ce6babe3e8f62) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240502 21:29:32.354822 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:32.475227 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:32.656283 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:32.974699 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:29:33.110530 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:33.139289 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:33.225100 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:33.228842 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:33.470482 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:33.555428 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:33.629688 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:29:33.666679 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(dc5c40e2bf0340958ecd3e7541209e95): perf score=0.033362
W20240502 21:29:33.704558 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:33.707247 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(dc5c40e2bf0340958ecd3e7541209e95) complete. Timing: real 0.040s user 0.028s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":427,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":2063,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2117,"threads_started":2,"wal-append.queue_time_us":1937}
I20240502 21:29:33.708426 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(dc5c40e2bf0340958ecd3e7541209e95): 4925 bytes on disk
W20240502 21:29:33.708492 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 21:29:33.709035 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(dc5c40e2bf0340958ecd3e7541209e95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240502 21:29:33.849251 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:33.918509 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:34.102909 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:34.112263 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(6c4aa176fd994cf0b55f304cf9d06fe0): perf score=0.033363
I20240502 21:29:34.147706 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.035s user 0.022s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":321,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1433,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":465,"threads_started":2,"wal-append.queue_time_us":352}
I20240502 21:29:34.149112 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0): 4925 bytes on disk
I20240502 21:29:34.149847 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 21:29:34.150928 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(ff7e019fa82d4648afaf130d3f6343d6): perf score=0.033372
I20240502 21:29:34.183591 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(ff7e019fa82d4648afaf130d3f6343d6) complete. Timing: real 0.032s user 0.026s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":311,"dirs.run_wall_time_us":1860,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":315,"threads_started":1,"wal-append.queue_time_us":479}
I20240502 21:29:34.184803 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(ff7e019fa82d4648afaf130d3f6343d6): 4585 bytes on disk
I20240502 21:29:34.185503 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(ff7e019fa82d4648afaf130d3f6343d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 21:29:34.211719 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(0fc717a1b12e4380a4107e6deae2b837): perf score=0.033397
I20240502 21:29:34.248261 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(0fc717a1b12e4380a4107e6deae2b837) complete. Timing: real 0.036s user 0.026s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":336,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1650,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":517,"threads_started":2,"wal-append.queue_time_us":498}
I20240502 21:29:34.249614 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(0fc717a1b12e4380a4107e6deae2b837): 4586 bytes on disk
I20240502 21:29:34.250232 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(0fc717a1b12e4380a4107e6deae2b837) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 21:29:34.314913 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:29:34.437171 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling FlushMRSOp(7a35079a106c4cca94cf65ffa93aefbf): perf score=0.033363
I20240502 21:29:34.488440 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: FlushMRSOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.051s user 0.044s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1372,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":275,"threads_started":1,"wal-append.queue_time_us":387}
I20240502 21:29:34.489377 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf): 4925 bytes on disk
I20240502 21:29:34.489912 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240502 21:29:34.640480 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:34.658416 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:34.874470 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(ac35b080037646b4b8b34c0eab2de4e6): perf score=0.033369
W20240502 21:29:34.905630 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 21:29:34.908308 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(ac35b080037646b4b8b34c0eab2de4e6) complete. Timing: real 0.034s user 0.023s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1021,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":1159,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":749,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":595,"threads_started":2,"wal-append.queue_time_us":371}
I20240502 21:29:34.909329 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(ac35b080037646b4b8b34c0eab2de4e6): 4840 bytes on disk
I20240502 21:29:34.909821 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(ac35b080037646b4b8b34c0eab2de4e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240502 21:29:35.139762 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:35.205662 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:35.518252 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:35.597853 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:29:35.669641 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:35.810266 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:35.842381 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:36.026414 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:36.133288 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:36.218502 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:29:36.219369 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:36.233891 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:36.270184 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:36.359516 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:29:36.453491 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0): perf score=0.033348
I20240502 21:29:36.485569 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0) complete. Timing: real 0.032s user 0.025s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":448,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1005,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":550,"threads_started":2,"wal-append.queue_time_us":390}
I20240502 21:29:36.486861 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0): 4586 bytes on disk
I20240502 21:29:36.487538 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(a8a9b14a7ff04ddab0d1e1a8e9fcb9b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240502 21:29:36.856539 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:36.876302 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:37.150985 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:29:37.192050 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(8f912eb05c574368909e0b3a8edf058d): perf score=0.033402
I20240502 21:29:37.227849 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(8f912eb05c574368909e0b3a8edf058d) complete. Timing: real 0.036s user 0.026s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1711,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":834,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":429,"threads_started":2,"wal-append.queue_time_us":343}
I20240502 21:29:37.229058 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(8f912eb05c574368909e0b3a8edf058d): 4925 bytes on disk
I20240502 21:29:37.229593 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(8f912eb05c574368909e0b3a8edf058d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240502 21:29:37.233989 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:37.322644 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 21:29:37.358318 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(7a35079a106c4cca94cf65ffa93aefbf): perf score=0.033379
I20240502 21:29:37.399961 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.041s user 0.032s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":341,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1029,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":436,"threads_started":2,"wal-append.queue_time_us":308}
I20240502 21:29:37.401124 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf): 4925 bytes on disk
I20240502 21:29:37.401680 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 21:29:37.402614 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(9963dcea478749e0a6e076fac6537fc4): perf score=0.033340
I20240502 21:29:37.446707 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(9963dcea478749e0a6e076fac6537fc4) complete. Timing: real 0.044s user 0.043s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1634,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":242,"threads_started":1,"wal-append.queue_time_us":383}
I20240502 21:29:37.447877 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(9963dcea478749e0a6e076fac6537fc4): 4586 bytes on disk
I20240502 21:29:37.448447 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(9963dcea478749e0a6e076fac6537fc4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240502 21:29:37.731016 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:37.815128 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:37.832448 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:29:37.875690 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:29:37.922330 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(bdf23b7a397441b5947ce6babe3e8f62): perf score=0.033393
I20240502 21:29:37.977135 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(bdf23b7a397441b5947ce6babe3e8f62) complete. Timing: real 0.054s user 0.039s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1536,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":827,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":5376,"thread_start_us":487,"threads_started":2,"wal-append.queue_time_us":381}
I20240502 21:29:37.978292 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(bdf23b7a397441b5947ce6babe3e8f62): 4840 bytes on disk
I20240502 21:29:37.978932 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(bdf23b7a397441b5947ce6babe3e8f62) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 21:29:38.113444 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:29:38.202108 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(9d8a25580bc74004b0e31c34f841d2cc): perf score=0.033350
W20240502 21:29:38.222326 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:38.233920 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:38.236567 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(9d8a25580bc74004b0e31c34f841d2cc) complete. Timing: real 0.034s user 0.027s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1606,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":481,"threads_started":2,"wal-append.queue_time_us":318}
I20240502 21:29:38.237788 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(9d8a25580bc74004b0e31c34f841d2cc): 4840 bytes on disk
I20240502 21:29:38.238464 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(9d8a25580bc74004b0e31c34f841d2cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 21:29:38.495525 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(566929e603d54be4bf931413f0fa1d7d): perf score=0.033354
I20240502 21:29:38.554602 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(566929e603d54be4bf931413f0fa1d7d) complete. Timing: real 0.059s user 0.046s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":482,"threads_started":2,"wal-append.queue_time_us":366}
I20240502 21:29:38.555663 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(566929e603d54be4bf931413f0fa1d7d): 4840 bytes on disk
I20240502 21:29:38.556224 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(566929e603d54be4bf931413f0fa1d7d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240502 21:29:38.650250 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:38.671386 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:38.735477 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:38.829874 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:38.837014 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:38.842741 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:39.257488 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:39.287485 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:39.396859 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:39.654235 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:39.870308 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:40.018126 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:29:40.027261 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:40.275826 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:40.306192 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:40.505404 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:40.563506 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:40.691169 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:40.695657 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:40.934371 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:41.060578 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:41.089109 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:41.269685 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:29:41.270606 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:41.296000 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:29:41.574187 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 21:29:41.644449 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:41.817950 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling FlushMRSOp(82fb425a5430426f8030ba5497736b11): perf score=0.033334
I20240502 21:29:41.854882 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: FlushMRSOp(82fb425a5430426f8030ba5497736b11) complete. Timing: real 0.037s user 0.036s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":325,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1907,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":728,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":424,"threads_started":2,"wal-append.queue_time_us":327}
I20240502 21:29:41.855826 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(82fb425a5430426f8030ba5497736b11): 4840 bytes on disk
I20240502 21:29:41.856340 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(82fb425a5430426f8030ba5497736b11) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240502 21:29:42.047246 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:42.081190 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:42.158732 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:42.449009 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:42.482409 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:42.484164 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:42.713047 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:29:43.007958 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling FlushMRSOp(98b5668e0d504db5a7874f041c14dbfa): perf score=0.033344
I20240502 21:29:43.034052 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling FlushMRSOp(9d8a25580bc74004b0e31c34f841d2cc): perf score=0.033394
I20240502 21:29:43.068919 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: FlushMRSOp(9d8a25580bc74004b0e31c34f841d2cc) complete. Timing: real 0.035s user 0.029s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":396,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1692,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":521,"threads_started":2,"wal-append.queue_time_us":347}
I20240502 21:29:43.069067 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: FlushMRSOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.061s user 0.052s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":388,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1793,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":453,"threads_started":2,"wal-append.queue_time_us":321}
I20240502 21:29:43.070286 26658 maintenance_manager.cc:419] P 9b0952b24bd8400c9940dbd5e28417b4: Scheduling UndoDeltaBlockGCOp(9d8a25580bc74004b0e31c34f841d2cc): 4840 bytes on disk
I20240502 21:29:43.070484 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa): 4840 bytes on disk
I20240502 21:29:43.070942 26547 maintenance_manager.cc:643] P 9b0952b24bd8400c9940dbd5e28417b4: UndoDeltaBlockGCOp(9d8a25580bc74004b0e31c34f841d2cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 21:29:43.070986 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240502 21:29:43.089306 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:43.095507 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:43.106372 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:29:43.251741 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling FlushMRSOp(049bec57d2474c328d2e26e16a4d33a9): perf score=0.033395
W20240502 21:29:43.253216 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 21:29:43.283676 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: FlushMRSOp(049bec57d2474c328d2e26e16a4d33a9) complete. Timing: real 0.031s user 0.026s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":1739,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":714,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":432,"threads_started":2,"wal-append.queue_time_us":337}
I20240502 21:29:43.284730 26391 maintenance_manager.cc:419] P b77f8f52372241d483f457389fda0f93: Scheduling UndoDeltaBlockGCOp(049bec57d2474c328d2e26e16a4d33a9): 4586 bytes on disk
I20240502 21:29:43.285251 26280 maintenance_manager.cc:643] P b77f8f52372241d483f457389fda0f93: UndoDeltaBlockGCOp(049bec57d2474c328d2e26e16a4d33a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240502 21:29:43.635903 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:43.673506 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:43.740505 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:43.762989 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:43.929498 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:43.965507 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:44.158908 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:29:44.252424 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling FlushMRSOp(7bf18ffb96d04d5e9ec8ee9563812ba0): perf score=0.033361
I20240502 21:29:44.298321 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: FlushMRSOp(7bf18ffb96d04d5e9ec8ee9563812ba0) complete. Timing: real 0.046s user 0.031s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3716,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":7138,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3906,"threads_started":2,"wal-append.queue_time_us":374}
I20240502 21:29:44.299417 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(7bf18ffb96d04d5e9ec8ee9563812ba0): 4925 bytes on disk
I20240502 21:29:44.299925 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(7bf18ffb96d04d5e9ec8ee9563812ba0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240502 21:29:44.430110 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling FlushMRSOp(2b6a4a74e0544e0e99e734a145c91d3b): perf score=0.033378
I20240502 21:29:44.477595 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: FlushMRSOp(2b6a4a74e0544e0e99e734a145c91d3b) complete. Timing: real 0.047s user 0.030s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":369,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1049,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":508,"threads_started":2,"wal-append.queue_time_us":374}
I20240502 21:29:44.478860 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(2b6a4a74e0544e0e99e734a145c91d3b): 4584 bytes on disk
I20240502 21:29:44.479423 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(2b6a4a74e0544e0e99e734a145c91d3b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240502 21:29:44.598305 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:44.627430 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:44.728016 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:44.752905 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:44.771611 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:45.023259 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:45.232196 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:29:45.360428 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:45.517467 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:45.547374 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:45.591110 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:45.794787 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:45.997143 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:46.128940 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:46.297765 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:29:46.333114 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:46.375758 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:46.435629 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:46.437507 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:46.882272 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:46.969516 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:47.169967 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:47.249194 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:47.253942 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:47.493120 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:47.615132 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:47.900240 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:47.912247 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:29:48.187815 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:48.204453 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:48.232896 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:48.357215 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:48.588155 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:48.696807 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:48.760859 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:48.810251 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:48.976662 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:49.198712 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:49.546178 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:49.668392 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:49.703505 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:49.729822 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:49.805474 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:29:50.029743 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:50.129444 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:50.448388 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:50.486680 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:50.612820 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:29:50.625041 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:50.688779 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:50.690855 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:50.773686 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:50.774873 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:50.998142 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:51.079533 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:51.289716 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:51.307205 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:51.314666 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:29:51.544067 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:51.724174 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:51.989243 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:52.013877 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:52.014806 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:52.221745 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:52.475759 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:52.513520 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:52.842614 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:52.888003 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:53.029006 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:53.062089 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:29:53.074476 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:53.281560 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:53.413676 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:53.447254 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:53.664299 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:53.839691 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:53.971789 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:54.089727 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:54.223278 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:54.502380 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:54.669240 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:54.966964 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:55.008745 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:55.009739 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:55.253510 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:55.271082 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:29:55.337759 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:55.531095 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:55.542912 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:55.825165 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:55.937085 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:56.058483 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:56.208561 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:29:56.209615 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:56.212644 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:56.257025 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:29:56.336555 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:29:56.589208 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:56.934883 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:57.024135 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:57.056115 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:29:57.076738 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:57.513439 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:57.585209 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:57.800912 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:57.967016 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:58.045871 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:29:58.046924 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:58.210604 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:29:58.351409 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:58.424422 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:58.511926 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:58.655791 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:58.781658 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:58.796134 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:58.853484 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:29:59.294947 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:29:59.411868 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:29:59.580468 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:29:59.701128 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:00.143443 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:00.173651 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:00.174580 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:00.464886 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:00.465947 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:00.697641 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:00.724570 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:00.726790 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:00.765538 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:30:00.933324 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:01.044148 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:01.133883 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:01.284087 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:01.333814 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:01.393170 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:30:01.422636 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:01.551805 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:01.894336 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:02.147887 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:02.149085 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:02.159523 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:02.727231 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:02.893322 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:02.921192 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:02.922899 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:03.166450 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:03.167454 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:03.293519 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:03.340085 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:03.441223 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:30:03.502054 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:03.740711 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:03.829142 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:03.931736 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:04.214044 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:04.455649 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:04.696158 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:04.775843 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:05.199007 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:05.403856 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:05.405007 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:05.484192 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:05.661846 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:05.668406 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:05.668511 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:05.741595 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:05.744081 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:05.900686 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:30:06.016507 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:06.065645 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:06.287868 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:06.430508 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:30:06.431416 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:06.545224 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:06.628371 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:06.999364 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:07.137092 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:07.150681 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:07.307297 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:07.478838 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:07.860427 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:07.880551 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:08.047638 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:08.093889 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:08.156067 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:08.264760 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:08.288986 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:30:08.442086 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:08.445256 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:08.554092 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:08.737015 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:08.742950 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:08.769495 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:09.174963 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:09.608100 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:09.684973 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:09.724581 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:10.262290 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:10.375419 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:10.589372 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:30:10.594465 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:10.638324 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:10.649343 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:10.750759 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:10.855530 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:10.932598 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:10.938555 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:10.941563 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:11.154012 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:11.225739 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:11.460557 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:30:11.463057 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:11.709508 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:11.834041 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:11.921795 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:12.326716 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:12.327775 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:12.333134 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:12.789891 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:12.794551 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:12.920612 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:12.938309 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:13.134073 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:30:13.242861 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:13.259927 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:13.331696 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:13.489058 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:13.511447 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:13.665577 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:13.751515 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:30:13.791499 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:14.093209 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:14.181957 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:14.573622 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:14.797183 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:15.025800 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:15.298874 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:15.405326 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:15.467696 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:15.527971 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:15.684345 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:30:15.725579 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:15.850577 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:15.899827 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:15.914868 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:15.932559 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:16.014928 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:16.113526 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:16.492902 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:30:16.495144 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:16.536139 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:16.809739 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:16.981880 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:17.244702 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:17.389556 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:17.456176 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:17.457168 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:17.720326 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:17.852139 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:17.899259 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:18.060619 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:30:18.129432 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:18.152693 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:18.166220 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:18.240521 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:18.259418 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:18.541885 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:18.787940 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:18.855165 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:19.164984 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:19.236881 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:19.275918 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:19.553093 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:19.993393 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:20.161064 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:20.230546 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:20.302634 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:20.304183 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:20.331686 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:20.503937 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:30:20.570878 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:20.660915 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:21.012302 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:21.019773 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:21.111883 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:21.177285 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:21.244114 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:21.589035 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:30:21.634280 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:21.664944 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:21.945667 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:22.244477 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:22.273195 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:22.483870 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:22.624919 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:30:22.626822 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:22.654867 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:22.788832 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:22.802105 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:22.829471 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:22.892860 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:30:23.180140 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:23.198333 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:23.280615 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:23.618012 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:23.758975 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:23.887249 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:24.135895 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:24.315274 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:24.363936 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:24.901340 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:24.928143 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:24.933631 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:25.227059 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:25.290163 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:25.338939 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:30:25.426775 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:25.599972 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:25.892372 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:25.944676 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:25.964665 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:26.341336 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:26.399348 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:26.509511 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:26.567994 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:26.569231 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:26.783143 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:30:26.833371 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:26.850476 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:27.456622 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:27.568737 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:27.725869 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:27.728888 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:27.730031 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:27.756991 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:27.843984 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:27.845330 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:30:27.945216 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:28.048848 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:28.242374 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:28.388875 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:28.477272 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:28.790740 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:28.836203 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:28.963611 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:29.001427 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:29.297786 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:29.315251 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:29.778498 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:30.052995 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:30.060649 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:30.171573 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:30.263757 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:30.420603 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:30:30.434773 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:30.503643 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:30.703886 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:30.704488 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:30.755018 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:31.278847 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:31.338900 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:31.700225 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:31.784850 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:30:31.790990 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:31.888237 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:31.903227 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:31.916956 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:32.313596 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:32.445940 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:32.496796 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:32.781513 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:32.904805 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:32.906082 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:32.906960 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:32.921700 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:32.923446 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:32.944386 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:30:33.293720 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:33.313751 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:33.403357 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:33.892541 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:34.142539 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:34.152758 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:34.299572 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:34.433043 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:34.452853 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:34.674474 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:34.984303 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:35.221094 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:35.230767 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:35.233628 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:30:35.251389 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:35.269763 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:35.568401 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:35.914263 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:36.087685 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:36.143002 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:36.442458 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:36.642692 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:36.774762 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:36.885248 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:30:36.886067 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:36.920568 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:37.078120 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:37.117946 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:37.296527 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:37.524417 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:37.685886 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:37.686833 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:37.761061 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:30:37.980144 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:37.981297 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:38.023917 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:38.057164 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:38.058097 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:38.461985 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:38.511027 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:38.519238 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:38.998873 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:39.060865 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:39.239820 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:39.241236 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:39.497502 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:39.636464 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:39.989965 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:40.065095 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:40.090262 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:40.253894 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:40.269155 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:40.444815 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:30:40.602723 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:40.633036 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:41.032941 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:41.092845 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:41.135382 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:41.645409 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:41.682973 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:41.824505 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:41.981041 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:30:42.148779 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:42.150943 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:42.259483 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:42.418002 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:42.528614 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:42.580134 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:42.733987 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:42.773428 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:42.835775 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:30:42.871824 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:43.116632 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:43.117506 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:43.356349 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:43.357277 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:43.578074 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:43.676898 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:43.783845 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:44.030819 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:44.175819 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:44.461455 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:44.515723 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:44.623423 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:44.716344 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:44.849620 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:45.011759 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:45.103930 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:45.233924 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:45.450762 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:30:45.474658 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:45.603471 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:45.924369 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:46.081089 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:46.290057 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:46.393342 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:46.427376 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:46.811897 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:47.003878 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:30:47.004755 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:47.019232 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:47.229214 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:47.328074 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:47.400842 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:47.420773 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:47.425876 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:47.540100 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:47.792382 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:47.923796 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:47.972635 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:30:48.021616 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:48.182390 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:48.183431 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:48.402282 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:48.403250 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:48.668334 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:48.711844 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:48.807354 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:49.296350 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:49.517541 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:49.543900 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:49.549991 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:49.627156 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:49.681897 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:49.863870 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:50.083501 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:50.182516 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:50.373895 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:50.470065 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:30:50.492380 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:50.676544 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:50.813115 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:50.814419 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:50.900305 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:51.227891 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:51.296697 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:51.312341 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:51.683363 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:52.027881 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 21:30:52.029016 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:52.032644 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:52.104163 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:52.222308 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:52.581243 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:52.601260 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:52.620496 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:52.624732 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:52.648885 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:53.007244 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:53.045888 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:53.071415 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:30:53.104765 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:53.470960 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:53.485980 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:53.488078 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:53.489259 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:53.667140 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:53.674327 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:54.245649 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:54.496934 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:54.542088 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:30:54.674587 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:54.702414 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:54.741940 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:55.050760 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:55.062207 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:55.218540 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:55.299670 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:55.450611 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:55.554489 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:30:55.705250 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:55.862680 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:55.903640 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:55.950871 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:30:55.994009 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:56.388059 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:56.812218 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:57.042456 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:30:57.121454 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:57.259171 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:57.381383 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:57.519418 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:57.733536 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:57.739959 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:30:57.790721 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:57.825366 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:57.965950 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:58.063609 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:58.261523 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:30:58.314657 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:58.341912 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:30:58.378556 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:58.503391 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:30:58.504065 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:58.523622 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:30:58.525023 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:30:58.860476 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:59.242192 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:30:59.255734 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:30:59.557288 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:30:59.685253 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:30:59.779359 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:00.121163 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:31:00.153846 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:31:00.177594 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:00.265435 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:00.702369 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:31:00.792565 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:00.806558 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:00.861943 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:00.894241 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:01.000828 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:01.004379 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:31:01.271044 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:01.659289 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:01.698115 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:02.007640 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:02.123376 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:31:02.240759 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:02.608615 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:02.635377 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:02.802112 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:02.802336 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:02.820405 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:03.103616 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:03.163909 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:03.238952 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:03.239421 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:03.269629 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:03.390578 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:03.409338 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:03.641734 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:31:03.670677 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:03.784821 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:03.879583 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:04.354125 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:04.457197 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:04.488258 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:04.775439 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:04.870008 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:04.940310 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:05.161705 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:05.325166 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:05.396721 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:05.674158 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:05.675720 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:05.763909 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:05.818213 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:05.942405 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:06.065191 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:31:06.066533 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:06.500372 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:06.811195 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:06.863304 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:07.066589 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:07.140734 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:31:07.471678 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:07.517477 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:07.894340 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:07.994297 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:08.027909 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:08.127696 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:08.248648 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:08.272333 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:08.273111 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:08.274405 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:08.288949 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:08.524911 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:08.715137 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:08.806418 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:08.867974 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:31:08.962224 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:09.181377 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:09.185840 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:09.305394 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:09.421308 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:09.920053 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:09.973126 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:10.086056 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:10.098572 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:10.335767 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:10.437932 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:10.503927 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:10.555109 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:10.641642 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:10.783458 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:10.921963 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:11.309468 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:11.448098 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:31:11.520543 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:11.717358 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:11.920643 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:12.032323 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:12.179519 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:31:12.189409 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:12.404973 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:12.453649 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:12.837347 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:12.882565 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:13.045629 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:13.114823 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:13.220834 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:13.272281 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:13.445060 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:13.446683 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:13.517325 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:13.765856 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:13.786437 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:14.216544 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:31:14.279292 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:14.400327 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:14.401456 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:14.557536 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:14.703099 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:14.845045 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:15.008267 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:15.028630 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:15.031203 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:15.232995 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:15.289814 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:15.622704 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:15.679643 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:15.917240 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:15.917690 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:15.948807 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:16.222092 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:16.334556 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:16.807780 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:31:16.814188 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:17.019100 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:17.127807 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:17.187844 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 21:31:17.320210 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:17.583300 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:17.767484 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:17.798753 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:17.822448 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:17.827615 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:18.212206 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:18.318074 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:18.395272 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:18.582968 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:18.693403 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:18.875804 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:18.936803 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:31:18.944096 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:19.020764 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:19.458791 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:19.460088 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:19.480770 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:19.548872 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:19.969719 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:20.017756 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:20.040638 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:20.223908 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:20.224964 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:20.245978 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:20.246845 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:20.482617 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:20.695847 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:20.813194 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:21.105520 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:21.306226 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:21.378589 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:31:21.390712 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:21.513556 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:21.892611 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:21.908138 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:22.036700 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:22.220506 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:31:22.388069 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:22.523435 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:22.532614 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:22.633846 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:22.654563 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:23.002189 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:23.035682 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:23.228147 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:23.603181 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:23.744040 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:23.760463 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:23.877663 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:31:23.940428 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:23.961967 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:24.136377 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:24.477005 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:24.514257 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:24.515363 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:24.524737 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:24.888131 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:25.054435 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:25.064685 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:25.085604 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:25.141973 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:25.466720 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:25.473318 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:25.527308 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:25.618386 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:25.632731 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:26.086959 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:26.166818 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:26.329983 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:31:26.376989 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:26.463690 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:26.778235 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:26.980367 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:27.143410 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:27.211251 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:27.511215 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:27.527285 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:31:27.553395 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:27.618722 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:27.728906 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:27.907158 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:27.965229 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:28.015233 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:28.302608 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:28.669627 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:28.800356 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:28.863706 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:29.001188 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:29.031580 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:31:29.153659 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:29.384699 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:29.392593 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:29.427412 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:29.705264 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:29.772078 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:29.772866 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:29.871378 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:29.961498 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:30.191861 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:30.287974 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:30.424474 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:30.425559 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:30.612198 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:30.727208 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:30.833851 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:31.224084 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:31.303959 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:31.321978 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:31:31.538975 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:31.804864 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:31.833436 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:31.969269 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:32.100378 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:32.235545 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:32.422842 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:32.455839 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:32.605547 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:31:32.623881 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:32.818768 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:32.856508 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:33.019575 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:33.342542 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:33.421134 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:33.726506 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:33.751374 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:31:33.802541 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:34.033635 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:34.183060 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:34.311319 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:34.392000 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:34.393334 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:34.446677 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:34.726002 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:34.745436 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:34.861054 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:34.861943 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:35.134227 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:35.148825 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:35.328045 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:35.684561 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:35.757196 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:35.783272 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:35.835937 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:35.927107 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:36.150069 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:36.193327 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:36.283277 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:31:36.641011 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:36.666118 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:36.951238 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:36.986856 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:37.009394 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:37.202471 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:37.421145 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:37.527117 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:37.653975 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:37.668582 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:37.735226 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:31:38.128654 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:38.158095 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:38.295047 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:38.430449 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:38.667194 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:38.774843 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:31:38.951916 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:39.232195 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:39.455361 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:39.456461 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:39.541033 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:39.576570 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:39.600145 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:39.825908 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:39.994267 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:40.078487 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:40.127918 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:40.208993 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:40.295823 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:40.452194 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:40.890134 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:40.964599 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:41.104430 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:41.171097 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:41.214857 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:31:41.270246 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:41.529167 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:41.621152 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:41.766851 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:41.916517 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:41.921650 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:41.967892 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:42.350582 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:42.378218 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:42.509652 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:42.549945 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:42.889333 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:42.914197 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:42.920094 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:31:43.352007 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:43.504943 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:43.571866 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:31:43.661437 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:43.833824 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:44.149163 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:44.354949 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:44.379834 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:44.445845 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:44.513628 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:44.627835 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:44.804950 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:44.835863 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:44.892915 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:31:44.893838 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:44.948434 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:45.260439 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:45.348657 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:45.349584 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:45.842413 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:46.173151 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:46.208086 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:31:46.380966 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:46.396495 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:46.437805 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:31:46.438715 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:46.588075 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:46.629160 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:46.883641 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:46.960680 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:46.990738 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:47.382037 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:47.402367 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:47.499524 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:47.585970 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:47.672784 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:47.863073 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:47.990177 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:31:48.397356 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:48.834156 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:48.864681 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:48.915473 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:48.934896 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:31:49.009029 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:49.211413 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:49.333213 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:49.489655 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:49.516100 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:49.877552 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:49.907398 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:49.990413 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:50.039165 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:50.079968 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:50.261775 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:50.263052 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:50.578840 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:31:50.579900 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:50.827246 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:50.828325 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:50.988502 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:51.161537 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:51.298879 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:51.299347 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:31:51.323287 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:51.475747 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:51.476583 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 21:31:51.759086 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:51.863619 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:51.872640 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:52.044124 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:52.100535 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:52.363165 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:52.422369 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:52.480950 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:52.550614 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:52.817484 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:53.015270 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:53.043298 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:31:53.423555 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:53.716965 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:53.725374 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:53.734088 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:53.888594 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:53.997403 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:31:54.344521 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:54.359885 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:54.484460 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:54.518168 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:54.592069 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:54.672930 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:54.910765 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:54.921979 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:55.293784 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:31:55.449334 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:55.450590 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:55.477691 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:55.713532 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:31:56.070729 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:56.230183 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:56.280328 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:56.284940 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:31:56.426134 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:31:56.467563 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:56.570636 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:31:56.937099 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:57.000717 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:57.044705 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:57.067097 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:57.091137 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 21:31:57.092312 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:57.265519 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:57.468458 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:57.505095 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:57.974162 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:57.986923 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:58.242647 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:31:58.352517 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:58.518200 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:58.571372 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:58.767119 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:31:58.838424 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:59.096807 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:31:59.317488 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:59.390221 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:31:59.499341 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:31:59.582460 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:31:59.720992 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:31:59.805536 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:31:59.824664 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:00.188459 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:00.231928 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:32:00.340027 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:00.622681 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:00.693168 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:00.930173 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:00.931231 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:01.292032 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:01.307677 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:32:01.435722 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:01.498920 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:32:01.631788 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:01.666865 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:01.823025 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:01.980363 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:02.080809 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:02.127470 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:02.258966 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:02.260162 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:02.578038 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:02.604864 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:02.658231 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:02.772467 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:03.184206 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:03.388024 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 21:32:03.531332 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:03.767436 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:04.070684 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:04.127244 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:04.186448 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:04.349959 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:32:04.572456 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:04.607604 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:04.618548 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:04.685647 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:04.873582 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:05.073348 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:05.114684 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:05.308827 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:05.388665 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:05.631263 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:05.932518 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:06.194650 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:06.245021 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:06.246265 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:06.324774 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:06.632071 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:06.670730 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:06.709151 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:32:06.814949 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:06.890682 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:06.891853 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:07.048837 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:07.300765 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 21:32:07.302047 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:07.406222 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:07.446241 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:07.674669 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:07.724184 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:07.825737 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:07.966634 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:08.422071 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:32:08.447607 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:08.709544 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:08.917682 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:08.941643 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:09.001943 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:09.184851 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:32:09.200951 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:09.608952 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:09.684538 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:09.805936 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:10.038022 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:10.078933 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:10.222849 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:10.267037 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:10.633332 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:10.690717 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:10.932240 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:10.942871 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:11.018918 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:11.368042 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:11.369424 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:11.481246 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:32:11.528779 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:11.530278 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:11.626672 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:11.673287 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:12.077250 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:12.090093 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:12.304970 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:12.305974 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:12.350086 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:12.396077 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:12.652629 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:12.659803 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:12.858772 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:13.044689 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:13.271122 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:13.451264 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:13.525749 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:13.526013 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:32:13.710759 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:14.023799 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:14.062410 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:14.137281 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:32:14.242064 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:14.613981 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:14.863124 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:14.936036 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:15.028551 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:15.060361 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:15.150662 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:15.238013 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:15.541224 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:15.605454 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:15.993539 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:16.086807 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:16.222869 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:16.297937 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:16.492391 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:32:16.565066 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:16.658772 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:16.659744 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:16.706846 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:16.959703 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:17.057410 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:17.362985 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:17.590942 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:17.630621 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:17.631814 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:17.651862 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:17.680334 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:17.738900 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:17.786608 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:18.040762 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:18.204959 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:18.586010 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:18.620560 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:32:18.664321 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:18.765093 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:19.096750 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:19.212738 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:19.232977 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:19.305905 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:32:19.500550 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:20.037266 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:20.065129 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:20.139206 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:20.240242 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:20.310652 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:20.524078 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:20.563781 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:20.605530 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:21.174194 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:21.176316 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:21.188599 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:21.604787 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:21.633680 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:21.674186 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:21.789477 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:21.790323 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:21.911970 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:32:22.042032 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:22.321280 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:22.510928 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:22.621287 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:22.651496 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:22.742645 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:22.750888 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:22.916839 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:22.927314 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:22.929891 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:22.984184 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:23.340729 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:23.575522 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:23.732389 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:23.738941 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:32:23.842725 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:24.212745 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:32:24.242338 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:24.330420 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:24.467433 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:24.723991 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:24.943519 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:25.195459 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:25.208091 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:25.324033 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:25.327436 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:25.361865 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:25.465718 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:25.739972 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:26.011945 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:26.181665 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:26.201149 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:26.583835 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:26.626832 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:26.809202 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:32:26.951545 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:26.964747 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:27.049897 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:27.169183 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:27.239755 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:27.474596 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:27.610381 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:27.634706 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:27.692569 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:27.695340 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:27.709578 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:27.852510 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:27.967463 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:28.074810 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:28.077138 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:28.302568 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:28.525400 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:28.619629 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:28.777093 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:32:28.995702 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:32:29.109879 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:29.161047 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:29.698251 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:29.703744 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:30.102412 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:30.154055 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:30.191993 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:30.205902 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:30.244892 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:30.297937 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:30.416528 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:30.595741 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:30.850735 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:30.944443 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:31.172792 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:31.213908 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:32:31.551467 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:31.646589 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:31.950269 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:31.969487 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:32.218287 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:32.219331 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:32.249982 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:32.400915 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:32.634627 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:32.689664 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:32.734846 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:32.764921 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:32.816968 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:32.827772 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:32.930006 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:33.157615 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:33.221277 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:33.345194 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:33.421999 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:33.667372 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:33.795043 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:32:34.041689 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:32:34.135350 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:34.277276 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:34.661015 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:34.673383 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:35.020936 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:35.128903 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:35.267391 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:35.437407 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:35.479559 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:35.532284 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:35.564421 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:35.680492 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:35.700357 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:35.861311 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:36.271755 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:36.612427 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:36.840281 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:32:36.853976 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:37.179720 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:37.370723 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 21:32:37.389750 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:32:37.393520 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:37.394717 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:37.479667 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:37.511713 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:37.773607 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:37.785392 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:37.897904 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:37.912187 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:37.968803 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:38.065274 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:38.169987 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:38.224546 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 21:32:38.327280 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:38.334676 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:38.805586 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:38.873544 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:38.925391 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:32:39.188917 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:32:39.464956 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:39.881608 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:39.930546 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:32:39.958807 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:39.959177 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:40.125653 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:40.180521 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:40.417080 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:40.434988 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:40.437319 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:40.446506 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:32:40.803642 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:40.825184 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:41.172587 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:32:41.213387 15162 consensus_queue.cc:579] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.001s)
W20240502 21:32:41.216076 24323 consensus_peers.cc:462] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:32:41.224421 24384 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:41.227388 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 21:32:41.227459 24783 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 21:32:41.228235 15347 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:32:41.228576 15162 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:32:41.231853 15346 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:32:41.232554 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:32:41.232643 26612 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:32:41.237071 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:32:41.237998 24245 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 14 to 15, NON_VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
W20240502 21:32:41.246299 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:41.247771 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:32:41.248255 24384 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:32:41.250597 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 21:32:41.250841 24783 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20240502 21:32:41.251359 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:41.251623 15346 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 21:32:41.252023 15162 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 21:32:41.254312 15346 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:41.254982 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:41.257860 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:41.258720 26612 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:41.260540 24248 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:32:41.653295 15378 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:32:41.654546 15378 tablet_copy_client.cc:308] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:32:41.660651 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 ({username='slave'} at 127.16.122.8:34017)
I20240502 21:32:41.660827 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 477d9df7d3a74b8584f3669be3256e00 from peer 1769ee000eb24298a367a0de1209e6e0 at {username='slave'} at 127.16.122.8:34017: session id = 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00
I20240502 21:32:41.664083 24408 tablet_copy_source_session.cc:205] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:32:41.665475 15378 ts_tablet_manager.cc:1875] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:32:41.669966 15378 ts_tablet_manager.cc:1888] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.12
I20240502 21:32:41.670176 15378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:32:41.672340 15378 tablet_copy_client.cc:783] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 5 data blocks...
I20240502 21:32:41.679117 15378 tablet_copy_client.cc:647] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Starting download of 1 WAL segments...
I20240502 21:32:41.681918 15378 tablet_copy_client.cc:523] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:32:41.685029 15378 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap starting.
I20240502 21:32:41.707370 15378 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:32:41.707815 15378 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Bootstrap complete.
I20240502 21:32:41.708128 15378 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.001s
I20240502 21:32:41.708726 15378 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:32:41.708932 15378 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1769ee000eb24298a367a0de1209e6e0, State: Initialized, Role: LEARNER
I20240502 21:32:41.709123 15378 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:32:41.709832 15378 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:32:41.710665 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 received from {username='slave'} at 127.16.122.8:34017
I20240502 21:32:41.710808 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1769ee000eb24298a367a0de1209e6e0-477d9df7d3a74b8584f3669be3256e00 on tablet 477d9df7d3a74b8584f3669be3256e00 with peer 1769ee000eb24298a367a0de1209e6e0
I20240502 21:32:41.834151 26792 maintenance_manager.cc:419] P 1769ee000eb24298a367a0de1209e6e0: Scheduling UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00): 4586 bytes on disk
I20240502 21:32:41.834903 26679 maintenance_manager.cc:643] P 1769ee000eb24298a367a0de1209e6e0: UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 21:32:41.947481 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:32:42.152779 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:32:42.220571 26741 raft_consensus.cc:1203] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
W20240502 21:32:42.318228 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:32:42.388520 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:32:42.414949 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:42.461851 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:32:42.600490 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 21:32:42.611925 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:32:42.714546 15162 raft_consensus.cc:1050] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 1769ee000eb24298a367a0de1209e6e0 to VOTER
I20240502 21:32:42.715296 15162 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } }
I20240502 21:32:42.718691 26740 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 21:32:42.718763 26612 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
W20240502 21:32:42.719333 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 21:32:42.719405 15414 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 21:32:42.720111 15414 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 21:32:42.720419 24783 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 21:32:42.721966 15413 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 21:32:42.723661 15243 consensus_queue.cc:579] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.015s)
I20240502 21:32:42.732388 15162 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:32:42.737674 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:32:42.743203 26741 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1769ee000eb24298a367a0de1209e6e0 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:32:42.748653 24245 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 16 to 17, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:32:42.758157 26610 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 9b0952b24bd8400c9940dbd5e28417b4 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } }
I20240502 21:32:42.758929 24783 consensus_queue.cc:237] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:32:42.770009 26344 raft_consensus.cc:1261] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 21:32:42.771318 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.771844 15465 consensus_queue.cc:1035] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 21:32:42.772388 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b2661e0b5fe4a0b94c150b14b8fd9ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.777019 15428 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:42.777743 26344 raft_consensus.cc:2932] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:42.781664 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7b2661e0b5fe4a0b94c150b14b8fd9ec with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:42.783718 24248 catalog_manager.cc:5484] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 9 to 10, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:42.795301 15135 consensus_queue.cc:579] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.166s)
W20240502 21:32:42.796684 26543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 21:32:42.804972 26610 consensus_queue.cc:237] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:42.807492 24783 raft_consensus.cc:1261] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer 9b0952b24bd8400c9940dbd5e28417b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
W20240502 21:32:42.807857 26542 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac35b080037646b4b8b34c0eab2de4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.808306 15470 consensus_queue.cc:1035] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240502 21:32:42.808800 26543 consensus_peers.cc:485] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.811113 15471 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:32:42.811836 24783 raft_consensus.cc:2932] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:32:42.814718 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ac35b080037646b4b8b34c0eab2de4e6 with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:42.816357 24246 catalog_manager.cc:5484] T ac35b080037646b4b8b34c0eab2de4e6 P 9b0952b24bd8400c9940dbd5e28417b4 reported cstate change: config changed from index 15 to 16, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 2 leader_uuid: "9b0952b24bd8400c9940dbd5e28417b4" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:32:42.835907 15400 consensus_queue.cc:579] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.205s)
W20240502 21:32:42.838259 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:32:42.845937 24519 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:32:42.848456 24648 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240502 21:32:42.848693 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9963dcea478749e0a6e076fac6537fc4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.849032 15475 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 21:32:42.849388 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.852165 15476 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:32:42.852783 24648 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:32:42.855785 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9963dcea478749e0a6e076fac6537fc4 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:42.857467 24248 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 12 to 13, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:32:42.871408 15400 consensus_queue.cc:579] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.219s)
I20240502 21:32:42.875203 24519 consensus_queue.cc:237] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:32:42.877833 24384 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:32:42.878008 24455 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ee3718b120e4fad9ae03e47d3f1fffa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.878576 15476 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:32:42.878859 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:42.881223 15400 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:42.881810 24384 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:42.884244 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2ee3718b120e4fad9ae03e47d3f1fffa with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:42.886555 24246 catalog_manager.cc:5484] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:32:42.985589 14066 consensus_queue.cc:579] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.358s)
W20240502 21:32:42.994123 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:32:42.998539 26344 consensus_queue.cc:237] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:43.002468 24783 raft_consensus.cc:1261] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Refusing update from remote peer b77f8f52372241d483f457389fda0f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240502 21:32:43.003551 14066 consensus_queue.cc:1035] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 21:32:43.004559 26277 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.008301 15488 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
W20240502 21:32:43.009168 26276 consensus_peers.cc:485] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93 -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdf23b7a397441b5947ce6babe3e8f62. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.009011 24783 raft_consensus.cc:2932] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:32:43.012729 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bdf23b7a397441b5947ce6babe3e8f62 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:43.015208 24246 catalog_manager.cc:5484] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 14 to 15, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 2 leader_uuid: "b77f8f52372241d483f457389fda0f93" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:32:43.054090 15121 consensus_queue.cc:579] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.276s)
I20240502 21:32:43.063444 24648 consensus_queue.cc:237] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:43.066481 24384 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:32:43.066594 24586 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cf38bbaaaa340f89d766a843c6bcaf9. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.066866 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.067193 15502 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:32:43.069741 15501 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:32:43.070487 24384 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } } }
I20240502 21:32:43.073486 24234 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:43.075393 24248 catalog_manager.cc:5484] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:32:43.193205 15475 consensus_queue.cc:579] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.561s)
I20240502 21:32:43.197289 24519 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:32:43.200070 26344 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:32:43.200707 15475 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 21:32:43.201056 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.201195 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.202659 15475 consensus_queue.cc:579] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.551s)
I20240502 21:32:43.203466 15475 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:32:43.204360 26344 raft_consensus.cc:2932] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:32:43.207172 24519 consensus_queue.cc:237] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
W20240502 21:32:43.210330 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 21:32:43.210366 24246 catalog_manager.cc:5484] T 2e5a87719c2544a5b0cb591ed9f46df8 P b77f8f52372241d483f457389fda0f93 reported cstate change: config changed from index 9 to 10, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:32:43.212430 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2e5a87719c2544a5b0cb591ed9f46df8 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:43.212567 24783 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:32:43.213490 24455 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c6e7eba855e48c0b0e18fa363ea01d4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.213966 15482 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:32:43.214284 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.232084 15400 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:43.236922 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:43.236805 24783 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
I20240502 21:32:43.239814 24246 catalog_manager.cc:5484] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 6 to 7, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:32:43.296985 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:32:43.303229 15521 ts_tablet_manager.cc:902] T 9963dcea478749e0a6e076fac6537fc4 P 9b0952b24bd8400c9940dbd5e28417b4: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
F20240502 21:32:43.303735 15521 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.queue_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.queue_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_cpu_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_cpu_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_wall_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_wall_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us) (tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.queue_time_us, tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.queue_time_us) (tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_cpu_time_us, tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_cpu_time_us) (tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_wall_time_us, tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_wall_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.queue_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.queue_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_cpu_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_cpu_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_wall_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_wall_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.queue_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.queue_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_cpu_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_cpu_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_wall_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_wall_time_us) (tablet-download-pool-5dd91818e74e42acb276a3864753094d.queue_time_us, tablet-download-pool-5dd91818e74e42acb276a3864753094d.queue_time_us) (tablet-download-pool-5dd91818e74e42acb276a3864753094d.run_cpu_time_us, tablet-download-pool-5dd91818e74e42acb276a3864753094d.run_cpu_time_us) (tablet-download-pool-5dd91818e74e42acb276a3864753094d.run_wall_time_us, tablet-download-pool-5dd91818e74e42acb276a3864753094d.run_wall_time_us) (tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.queue_time_us, tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.queue_time_us) (tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_cpu_time_us, tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_cpu_time_us) (tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_wall_time_us, tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_wall_time_us) (tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.queue_time_us, tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.queue_time_us) (tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.run_cpu_time_us, tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.run_cpu_time_us) (tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.run_wall_time_us, tablet-download-pool-855b247f654d41c2bbb13abb25d35c1c.run_wall_time_us) (tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.queue_time_us, tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.queue_time_us) (tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_cpu_time_us, tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_cpu_time_us) (tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_wall_time_us, tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_wall_time_us) (tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.queue_time_us, tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.queue_time_us) (tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_cpu_time_us, tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_cpu_time_us) (tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_wall_time_us, tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_wall_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us) (tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.queue_time_us, tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.queue_time_us) (tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_cpu_time_us, tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_cpu_time_us) (tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_wall_time_us, tablet-download-pool-9d8a25580bc74004b0e31c34f841d2cc.run_wall_time_us) (tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.queue_time_us, tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.queue_time_us) (tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_cpu_time_us, tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_cpu_time_us) (tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_wall_time_us, tablet-download-pool-ac35b080037646b4b8b34c0eab2de4e6.run_wall_time_us) (tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.queue_time_us, tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.queue_time_us) (tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_cpu_time_us, tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_cpu_time_us) (tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_wall_time_us, tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_wall_time_us) (tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.queue_time_us, tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.queue_time_us) (tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_cpu_time_us, tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_cpu_time_us) (tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_wall_time_us, tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4b0d822dcd google::LogMessage::Fail() at ??:0
@ 0x7f4b0d826b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4b0d8227cc google::LogMessage::Flush() at ??:0
@ 0x7f4b0d823f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4b10563742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4b1052ff8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4b10529cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f4b1f97c8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f4b1f994daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f4b1fabf480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f4b1faf7a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f4b1fada981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f4b1fada70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4b10533b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4b1053719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 21:32:43.360844 15528 ts_tablet_manager.cc:902] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
F20240502 21:32:43.361346 15528 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.queue_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.queue_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_cpu_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_cpu_time_us) (tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_wall_time_us, tablet-download-pool-477d9df7d3a74b8584f3669be3256e00.run_wall_time_us) (tablet-download-pool-53c1992a21224bac883da474f143c1d7.queue_time_us, tablet-download-pool-53c1992a21224bac883da474f143c1d7.queue_time_us) (tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_cpu_time_us, tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_cpu_time_us) (tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_wall_time_us, tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_wall_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.queue_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.queue_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_cpu_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_cpu_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_wall_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_wall_time_us) (tablet-download-pool-616e911db4384c17905713adc595ea0a.queue_time_us, tablet-download-pool-616e911db4384c17905713adc595ea0a.queue_time_us) (tablet-download-pool-616e911db4384c17905713adc595ea0a.run_cpu_time_us, tablet-download-pool-616e911db4384c17905713adc595ea0a.run_cpu_time_us) (tablet-download-pool-616e911db4384c17905713adc595ea0a.run_wall_time_us, tablet-download-pool-616e911db4384c17905713adc595ea0a.run_wall_time_us) (tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.queue_time_us, tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.queue_time_us) (tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_cpu_time_us, tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_cpu_time_us) (tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_wall_time_us, tablet-download-pool-6c4aa176fd994cf0b55f304cf9d06fe0.run_wall_time_us) (tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.queue_time_us, tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.queue_time_us) (tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_cpu_time_us, tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_cpu_time_us) (tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_wall_time_us, tablet-download-pool-7a35079a106c4cca94cf65ffa93aefbf.run_wall_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.queue_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.queue_time_us) (tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.queue_time_us, tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.queue_time_us) (tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_cpu_time_us, tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_cpu_time_us) (tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_wall_time_us, tablet-download-pool-7bf18ffb96d04d5e9ec8ee9563812ba0.run_wall_time_us) (tablet-download-pool-82fb425a5430426f8030ba5497736b11.queue_time_us, tablet-download-pool-82fb425a5430426f8030ba5497736b11.queue_time_us) (tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_cpu_time_us, tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_cpu_time_us) (tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_wall_time_us, tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_wall_time_us) (tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.queue_time_us, tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.queue_time_us) (tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_cpu_time_us, tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_cpu_time_us) (tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_wall_time_us, tablet-download-pool-98b5668e0d504db5a7874f041c14dbfa.run_wall_time_us) (tablet-download-pool-adde18faca794600bb4aa24f0cf744e6.queue_time_us, tablet-download-pool-adde18faca794600bb4aa24f0cf744e6.queue_time_us) (tablet-download-pool-adde18faca794600bb4aa24f0cf744e6.run_cpu_time_us, tablet-download-pool-adde18faca794600bb4aa24f0cf744e6.run_cpu_time_us) (tablet-download-pool-adde18faca794600bb4aa24f0cf744e6.run_wall_time_us, tablet-download-pool-adde18faca794600bb4aa24f0cf744e6.run_wall_time_us) (tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.queue_time_us, tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.queue_time_us) (tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_cpu_time_us, tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_cpu_time_us) (tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_wall_time_us, tablet-download-pool-dc5c40e2bf0340958ecd3e7541209e95.run_wall_time_us) (tablet-download-pool-e847a796029d4edea79694f5402a291a.queue_time_us, tablet-download-pool-e847a796029d4edea79694f5402a291a.queue_time_us) (tablet-download-pool-e847a796029d4edea79694f5402a291a.run_cpu_time_us, tablet-download-pool-e847a796029d4edea79694f5402a291a.run_cpu_time_us) (tablet-download-pool-e847a796029d4edea79694f5402a291a.run_wall_time_us, tablet-download-pool-e847a796029d4edea79694f5402a291a.run_wall_time_us) (tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.queue_time_us, tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.queue_time_us) (tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_cpu_time_us, tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_cpu_time_us) (tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_wall_time_us, tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
@ 0x7f4b10537019 std::_Function_handler<>::_M_invoke() at ??:0
*** Check failure stack trace: ***
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb5bedd3dcd google::LogMessage::Fail() at ??:0
@ 0x7fb5bedd7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4b1051201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb5bedd37cc google::LogMessage::Flush() at ??:0
@ 0x7fb5bedd4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4b138976db start_thread at ??:0
@ 0x7f4b0c6d571f clone at ??:0
@ 0x7fb5c1b14742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fb5c1ae0f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fb5c1adacd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fb5d0f2d8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fb5d0f45daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fb5d1070480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fb5d10a8a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fb5d108b981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fb5d108b70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb5c1ae4b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb5c1ae819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb5c1ae8019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 21:32:43.428334 15539 ts_tablet_manager.cc:902] T 2ee3718b120e4fad9ae03e47d3f1fffa P 1769ee000eb24298a367a0de1209e6e0: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
@ 0x7fb5c1ac301e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb5c4e486db start_thread at ??:0
@ 0x7fb5bdc8671f clone at ??:0
W20240502 21:32:43.460239 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:32:43.492020 15553 ts_tablet_manager.cc:902] T 4cf38bbaaaa340f89d766a843c6bcaf9 P b77f8f52372241d483f457389fda0f93: Initiating tablet copy from peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427)
F20240502 21:32:43.492493 15553 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.queue_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.queue_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_cpu_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_cpu_time_us) (tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_wall_time_us, tablet-download-pool-049bec57d2474c328d2e26e16a4d33a9.run_wall_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.queue_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_cpu_time_us) (tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us, tablet-download-pool-0fc717a1b12e4380a4107e6deae2b837.run_wall_time_us) (tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.queue_time_us, tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.queue_time_us) (tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_cpu_time_us, tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_cpu_time_us) (tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_wall_time_us, tablet-download-pool-2b6a4a74e0544e0e99e734a145c91d3b.run_wall_time_us) (tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.queue_time_us, tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.queue_time_us) (tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_cpu_time_us, tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_cpu_time_us) (tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_wall_time_us, tablet-download-pool-2e5a87719c2544a5b0cb591ed9f46df8.run_wall_time_us) (tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.queue_time_us, tablet-download-pool-4cf38bbaaaa340f89d766a843c6bcaf9.queue_time_us) (tablet-download-pool-53c1992a21224bac883da474f143c1d7.queue_time_us, tablet-download-pool-53c1992a21224bac883da474f143c1d7.queue_time_us) (tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_cpu_time_us, tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_cpu_time_us) (tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_wall_time_us, tablet-download-pool-53c1992a21224bac883da474f143c1d7.run_wall_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.queue_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.queue_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_cpu_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_cpu_time_us) (tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_wall_time_us, tablet-download-pool-566929e603d54be4bf931413f0fa1d7d.run_wall_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.queue_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.queue_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_cpu_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_cpu_time_us) (tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_wall_time_us, tablet-download-pool-7b2661e0b5fe4a0b94c150b14b8fd9ec.run_wall_time_us) (tablet-download-pool-82fb425a5430426f8030ba5497736b11.queue_time_us, tablet-download-pool-82fb425a5430426f8030ba5497736b11.queue_time_us) (tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_cpu_time_us, tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_cpu_time_us) (tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_wall_time_us, tablet-download-pool-82fb425a5430426f8030ba5497736b11.run_wall_time_us) (tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.queue_time_us, tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.queue_time_us) (tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_cpu_time_us, tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_cpu_time_us) (tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_wall_time_us, tablet-download-pool-8f912eb05c574368909e0b3a8edf058d.run_wall_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.queue_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_cpu_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_cpu_time_us) (tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_wall_time_us, tablet-download-pool-9963dcea478749e0a6e076fac6537fc4.run_wall_time_us) (tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.queue_time_us, tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.queue_time_us) (tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_cpu_time_us, tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_cpu_time_us) (tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_wall_time_us, tablet-download-pool-bdf23b7a397441b5947ce6babe3e8f62.run_wall_time_us) (tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.queue_time_us, tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.queue_time_us) (tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_cpu_time_us, tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_cpu_time_us) (tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_wall_time_us, tablet-download-pool-d0ba1af1d3624d6387bf892873f3f67e.run_wall_time_us) (tablet-download-pool-e847a796029d4edea79694f5402a291a.queue_time_us, tablet-download-pool-e847a796029d4edea79694f5402a291a.queue_time_us) (tablet-download-pool-e847a796029d4edea79694f5402a291a.run_cpu_time_us, tablet-download-pool-e847a796029d4edea79694f5402a291a.run_cpu_time_us) (tablet-download-pool-e847a796029d4edea79694f5402a291a.run_wall_time_us, tablet-download-pool-e847a796029d4edea79694f5402a291a.run_wall_time_us) (tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.queue_time_us, tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.queue_time_us) (tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_cpu_time_us, tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_cpu_time_us) (tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_wall_time_us, tablet-download-pool-ff7e019fa82d4648afaf130d3f6343d6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f79b4cd3dcd google::LogMessage::Fail() at ??:0
@ 0x7f79b4cd7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f79b4cd37cc google::LogMessage::Flush() at ??:0
@ 0x7f79b4cd4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f79b7a14742 kudu::TraceMetrics::InternName() at ??:0
I20240502 21:32:43.526898 15559 ts_tablet_manager.cc:902] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
@ 0x7f79b79e0f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f79b79dacd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f79c6e2d8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f79c6e45daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f79c6f70480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 21:32:43.554404 15559 tablet_copy_client.cc:308] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.5:42587
I20240502 21:32:43.555374 26365 tablet_copy_service.cc:140] P b77f8f52372241d483f457389fda0f93: Received BeginTabletCopySession request for tablet bdf23b7a397441b5947ce6babe3e8f62 from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:33885)
I20240502 21:32:43.555565 26365 tablet_copy_service.cc:161] P b77f8f52372241d483f457389fda0f93: Beginning new tablet copy session on tablet bdf23b7a397441b5947ce6babe3e8f62 from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:33885: session id = 7f8bb40905da4891b461220b95c66832-bdf23b7a397441b5947ce6babe3e8f62
@ 0x7f79c6fa8a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 21:32:43.560544 26365 tablet_copy_source_session.cc:205] T bdf23b7a397441b5947ce6babe3e8f62 P b77f8f52372241d483f457389fda0f93: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:32:43.562610 15559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf23b7a397441b5947ce6babe3e8f62. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f79c6f8b981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f79c6f8b70e std::_Function_handler<>::_M_invoke() at ??:0
W20240502 21:32:43.569358 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 22 similar messages]
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 21:32:43.573796 15559 tablet_copy_client.cc:783] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 5 data blocks...
@ 0x7f79b79e4b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f79b79e819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f79b79e8019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f79b79c301e kudu::Thread::SuperviseThread() at ??:0
I20240502 21:32:43.590319 15559 tablet_copy_client.cc:647] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
@ 0x7f79bad486db start_thread at ??:0
I20240502 21:32:43.594000 15559 tablet_copy_client.cc:523] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f79b3b8671f clone at ??:0
I20240502 21:32:43.599964 15559 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:32:43.644124 15559 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:32:43.644582 15559 tablet_bootstrap.cc:492] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:32:43.644976 15559 ts_tablet_manager.cc:1367] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.005s
I20240502 21:32:43.645808 15559 raft_consensus.cc:348] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:43.646152 15559 raft_consensus.cc:729] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:32:43.654516 15559 consensus_queue.cc:260] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:43.655244 15559 ts_tablet_manager.cc:1398] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:32:43.697325 15586 ts_tablet_manager.cc:902] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:32:43.710093 15586 tablet_copy_client.cc:308] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:32:43.719033 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(bdf23b7a397441b5947ce6babe3e8f62): 4840 bytes on disk
I20240502 21:32:43.719745 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(bdf23b7a397441b5947ce6babe3e8f62) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240502 21:32:43.745926 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 2e5a87719c2544a5b0cb591ed9f46df8 from peer 1398472b92c842f285b8c7cc43cc1527 ({username='slave'} at 127.16.122.4:38439)
I20240502 21:32:43.746119 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 2e5a87719c2544a5b0cb591ed9f46df8 from peer 1398472b92c842f285b8c7cc43cc1527 at {username='slave'} at 127.16.122.4:38439: session id = 1398472b92c842f285b8c7cc43cc1527-2e5a87719c2544a5b0cb591ed9f46df8
I20240502 21:32:43.749773 24541 tablet_copy_source_session.cc:205] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:32:43.754971 15586 ts_tablet_manager.cc:1875] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:32:43.764745 15586 ts_tablet_manager.cc:1888] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 21:32:43.765048 15586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5a87719c2544a5b0cb591ed9f46df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:32:43.768419 15586 tablet_copy_client.cc:783] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 5 data blocks...
I20240502 21:32:43.776086 15586 tablet_copy_client.cc:647] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 1 WAL segments...
I20240502 21:32:43.778245 15586 tablet_copy_client.cc:523] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:32:43.783272 15586 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:32:43.801757 15457 consensus_queue.cc:579] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.463s)
I20240502 21:32:43.807104 24384 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:43.842882 24648 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 21:32:43.843590 15599 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 21:32:43.865032 15457 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:32:43.869824 24649 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:32:43.870787 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:32:43.871896 24246 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 14 to 15, VOTER 836775e329f148e5908a1afc54f06428 (127.16.122.6) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:32:43.886500 15586 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:32:43.887099 15586 tablet_bootstrap.cc:492] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Bootstrap complete.
I20240502 21:32:43.887434 15586 ts_tablet_manager.cc:1367] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.104s user 0.043s sys 0.021s
I20240502 21:32:43.888342 15586 raft_consensus.cc:348] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:32:43.888685 15586 raft_consensus.cc:729] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: LEARNER
I20240502 21:32:43.888895 15586 consensus_queue.cc:260] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:32:43.889642 15586 ts_tablet_manager.cc:1398] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:32:43.892701 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 1398472b92c842f285b8c7cc43cc1527-2e5a87719c2544a5b0cb591ed9f46df8 received from {username='slave'} at 127.16.122.4:38439
I20240502 21:32:43.892884 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 1398472b92c842f285b8c7cc43cc1527-2e5a87719c2544a5b0cb591ed9f46df8 on tablet 2e5a87719c2544a5b0cb591ed9f46df8 with peer 1398472b92c842f285b8c7cc43cc1527
I20240502 21:32:43.912277 24384 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }
I20240502 21:32:43.914736 24649 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 21:32:43.915307 15457 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 21:32:43.917845 15457 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
W20240502 21:32:43.920912 24323 connection.cc:535] client connection to 127.16.122.7:36311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:43.921106 24456 connection.cc:535] client connection to 127.16.122.7:36311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:32:43.922518 24648 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
W20240502 21:32:43.925522 24587 connection.cc:535] client connection to 127.16.122.7:36311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:43.927147 15600 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:32:43.921800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.7:36311 (local address 127.16.122.1:55053)
0502 21:32:43.922018 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0502 21:32:43.922037 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0502 21:32:43.922172 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:32:43.926865 (+ 4693us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118}
W20240502 21:32:43.927351 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.927464 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.927534 31006 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:32:43.921806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.7:36311 (local address 127.16.122.2:45497)
0502 21:32:43.921959 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0502 21:32:43.921981 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0502 21:32:43.922148 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:32:43.926833 (+ 4685us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
W20240502 21:32:43.927568 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.927649 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.927700 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:43.927815 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.927924 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.928157 24455 connection.cc:535] client connection to 127.16.122.8:37045 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:43.928292 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 57 similar messages]
W20240502 21:32:43.928699 24717 connection.cc:535] client connection to 127.16.122.8:37045 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:43.928787 14718 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:32:43.926409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.7:36311 (local address 127.16.122.3:41853)
0502 21:32:43.926871 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0502 21:32:43.926892 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0502 21:32:43.927035 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:32:43.928252 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":329}
W20240502 21:32:43.928975 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.929824 24322 connection.cc:535] client connection to 127.16.122.8:37045 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:43.931419 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:43.931540 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.931641 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:43.931640 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 21:32:43.931716 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.931813 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.932201 24248 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 15 to 16, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } }
W20240502 21:32:43.932720 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.932824 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:43.932904 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.932996 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.939167 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.939296 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.939364 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.939422 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.946038 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:43.951884 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(2e5a87719c2544a5b0cb591ed9f46df8): 4586 bytes on disk
I20240502 21:32:43.952445 15501 consensus_queue.cc:579] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.101s)
I20240502 21:32:43.952790 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(2e5a87719c2544a5b0cb591ed9f46df8) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
I20240502 21:32:43.962008 24648 consensus_queue.cc:237] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:32:43.973444 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:43.984709 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:44.049311 24586 connection.cc:535] client connection to 127.16.122.5:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:44.049404 24455 connection.cc:535] client connection to 127.16.122.5:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:44.049944 15559 tablet_copy_client.cc:1108] T bdf23b7a397441b5947ce6babe3e8f62 P 7f8bb40905da4891b461220b95c66832: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:44.049974 24717 connection.cc:535] client connection to 127.16.122.5:42587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:32:44.050856 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:44.051091 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.051211 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.051733 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.051880 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.051956 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.051990 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.052086 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.099766 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.116640 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:44.119133 24778 raft_consensus.cc:1203] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240502 21:32:44.136763 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:44.190807 15627 consensus_queue.cc:579] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.034s)
W20240502 21:32:44.194430 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:32:44.195464 24519 consensus_queue.cc:237] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } }
I20240502 21:32:44.199810 24648 raft_consensus.cc:1261] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 3 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 21:32:44.200495 15628 consensus_queue.cc:1035] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 21:32:44.200940 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:44.202319 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.202543 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:44.203162 15628 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:32:44.203770 24648 raft_consensus.cc:2932] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 427345e9a65d417c84a773ac1925a18a [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } } }
I20240502 21:32:44.206600 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:32:44.207549 24248 catalog_manager.cc:5484] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 10 to 12, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) added. New cstate: current_term: 3 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: NON_VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:32:44.414680 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:44.503392 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:44.523798 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:44.526187 15628 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:44.526408 15628 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:32:44.527084 15628 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:44.528502 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:44.528643 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:44.528774 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:44.529172 15628 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:44.529830 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:44.539575 15632 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:44.539790 15632 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:44.540572 15632 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:44.542260 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:44.542435 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:44.542548 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:32:44.542778 15632 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:44.612983 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:44.625346 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:44.652375 15400 raft_consensus.cc:1050] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832: attempting to promote NON_VOTER 1398472b92c842f285b8c7cc43cc1527 to VOTER
I20240502 21:32:44.653278 15400 consensus_queue.cc:237] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } }
I20240502 21:32:44.656343 24778 raft_consensus.cc:1261] T 2e5a87719c2544a5b0cb591ed9f46df8 P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:32:44.657331 15627 consensus_queue.cc:1035] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:32:44.657557 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:44.657896 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:32:44.811434 15428 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:44.811684 15428 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:44.812430 15428 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:44.813853 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:44.814026 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:44.814117 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:44.814409 15428 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:44.882107 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:44.937758 15428 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:44.937999 15428 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:44.938845 15428 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:32:44.940218 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:44.940402 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:32:44.940550 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:32:44.940815 15428 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:44.986658 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:44.992205 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:45.005502 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:32:45.022073 15457 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:45.022315 15457 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:32:45.023136 15457 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
I20240502 21:32:45.023756 24778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "e856847c3f494e6aa57f9cecf117ce4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "1398472b92c842f285b8c7cc43cc1527" is_pre_election: true
W20240502 21:32:45.024609 24322 leader_election.cc:334] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:45.024727 24322 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e856847c3f494e6aa57f9cecf117ce4d; no voters: 1398472b92c842f285b8c7cc43cc1527, b77f8f52372241d483f457389fda0f93
I20240502 21:32:45.024968 15457 raft_consensus.cc:2726] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:32:45.036535 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:45.093771 15428 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:45.093981 15428 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:32:45.094784 15428 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
W20240502 21:32:45.096285 24717 leader_election.cc:334] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:45.100533 24384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d" is_pre_election: true
I20240502 21:32:45.100764 24384 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 3.
I20240502 21:32:45.101202 24718 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, e856847c3f494e6aa57f9cecf117ce4d; no voters: b77f8f52372241d483f457389fda0f93
I20240502 21:32:45.101516 15428 raft_consensus.cc:2781] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 21:32:45.101620 15428 raft_consensus.cc:482] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:45.101733 15428 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:32:45.103896 15428 raft_consensus.cc:504] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:32:45.104496 15428 leader_election.cc:288] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 4 election: Requested vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:32:45.105129 24384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6a4a74e0544e0e99e734a145c91d3b" candidate_uuid: "1398472b92c842f285b8c7cc43cc1527" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "e856847c3f494e6aa57f9cecf117ce4d"
I20240502 21:32:45.105362 24384 raft_consensus.cc:3037] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 3 FOLLOWER]: Advancing to term 4
W20240502 21:32:45.105962 24717 leader_election.cc:334] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:45.107504 24384 raft_consensus.cc:2454] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1398472b92c842f285b8c7cc43cc1527 in term 4.
I20240502 21:32:45.107976 24718 leader_election.cc:302] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527, e856847c3f494e6aa57f9cecf117ce4d; no voters: b77f8f52372241d483f457389fda0f93
I20240502 21:32:45.108300 15428 raft_consensus.cc:2781] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:32:45.108724 15428 raft_consensus.cc:686] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [term 4 LEADER]: Becoming Leader. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Running, Role: LEADER
I20240502 21:32:45.109022 15428 consensus_queue.cc:237] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } }
I20240502 21:32:45.111932 24248 catalog_manager.cc:5484] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: term changed from 3 to 4, leader changed from b77f8f52372241d483f457389fda0f93 (127.16.122.5) to 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4). New cstate: current_term: 4 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 21:32:45.178102 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:45.224771 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:45.394882 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:45.460810 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:45.522893 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:45.546603 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:45.566053 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:45.572698 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:45.580119 24384 raft_consensus.cc:1261] T 2b6a4a74e0544e0e99e734a145c91d3b P e856847c3f494e6aa57f9cecf117ce4d [term 4 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240502 21:32:45.580791 15428 consensus_queue.cc:1035] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 21:32:45.581720 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:45.587003 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:32:45.692009 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:45.798735 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:45.906448 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:45.955158 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:45.956883 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:45.966398 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:46.084420 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:46.085430 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.100780 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.148686 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.171198 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:46.248907 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.272691 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:32:46.278569 15699 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:46.278769 15699 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:32:46.279632 15699 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:46.281111 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:46.281289 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:46.281394 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:46.281651 15699 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:46.301829 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:32:46.306668 15428 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:46.306872 15428 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:46.307564 15428 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:46.308967 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:46.309129 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:46.309248 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:32:46.309473 15428 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:46.314882 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.326900 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.342197 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:46.342614 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:32:46.366319 15428 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:46.366587 15428 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:46.367439 15428 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:46.368045 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.368647 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:46.368813 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:46.368940 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:46.369192 15428 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:46.417330 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.423498 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.442749 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.447212 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.477584 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:46.477741 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:46.483819 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:32:46.505236 24237 maintenance_manager.cc:419] P 34199807c207492781e8dba15a3f67ab: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 12788 bytes on disk
I20240502 21:32:46.510109 24236 maintenance_manager.cc:643] P 34199807c207492781e8dba15a3f67ab: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":364,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":129,"thread_start_us":532,"threads_started":2,"wal-append.queue_time_us":380}
W20240502 21:32:46.512372 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.538650 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.554095 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.562506 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.564159 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:46.581194 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.624003 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:32:46.628389 15428 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:46.628624 15428 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:46.629354 15428 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:32:46.630656 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:46.631707 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:46.631839 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:32:46.632261 15428 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:46.650674 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.677948 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:46.695852 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.732504 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.736224 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.764019 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:46.882719 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:46.994467 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:47.039939 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:47.056075 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:47.094266 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:47.107023 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:47.132572 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:47.368141 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:47.467730 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:47.472627 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:47.506673 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:47.539573 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:47.579288 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:47.714181 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:47.767464 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 21:32:47.784341 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:32:47.786325 15699 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:47.786568 15699 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:32:47.787191 15699 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:47.788568 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:47.788760 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:47.788916 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:47.789193 15699 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:47.912814 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:47.923751 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.013839 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.046262 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.138484 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.158600 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:48.165971 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 21:32:48.167348 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:32:48.302996 15428 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:48.303231 15428 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:48.303947 15428 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:48.304595 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:32:48.305645 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:48.305800 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:48.305905 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:32:48.306169 15428 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:48.342295 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:32:48.385133 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.411943 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.441613 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.444597 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.480518 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:32:48.508085 15879 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:48.508348 15879 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:48.509119 15879 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:32:48.510537 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:48.510702 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:48.510852 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:32:48.511111 15879 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:48.512598 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.521013 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:48.531551 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.533593 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.536058 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.537700 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.736941 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.750424 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.780560 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:48.781360 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 21:32:48.830976 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.833204 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.852699 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.884393 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.892859 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.917326 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:48.918064 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.953331 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 21:32:48.954317 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:48.957206 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.988248 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:48.988914 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:49.050117 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.059985 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.072860 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.091197 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.105515 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:49.133105 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.156236 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.168659 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:49.180805 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.276291 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.299454 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.323328 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:32:49.371060 15879 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:49.371294 15879 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:49.372015 15879 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:49.373770 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:49.373935 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:49.374012 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:49.374258 15879 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:49.383623 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.388617 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.396618 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:49.483433 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:49.544366 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:49.551687 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:49.562510 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:49.651609 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:49.822732 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:49.942377 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:49.974547 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.048643 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:50.048844 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.090402 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.104163 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.180941 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.240000 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:32:50.278487 15879 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:50.278723 15879 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:50.279584 15879 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:50.281100 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:50.281280 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:50.281402 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:32:50.281677 15879 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:50.394752 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:50.397125 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.559587 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:50.587673 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.672575 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.697999 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.701772 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.799865 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.826887 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:50.840265 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.849663 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:50.858335 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:50.915330 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:50.945564 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:32:50.947064 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.119661 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:51.131767 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.136977 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.158242 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.179056 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.181857 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:51.309396 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.312410 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:51.337265 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.398479 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.399224 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.404436 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.441498 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:51.460980 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:51.466742 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.482632 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.488281 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.515913 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.575134 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.620112 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.622578 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:51.628015 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:51.688349 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.705266 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.723834 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.727941 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.749501 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.752583 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:51.786737 16049 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:51.786957 16049 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:51.787720 16049 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:51.789531 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:51.789800 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:51.790035 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:32:51.790390 16049 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:51.820473 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.826514 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:32:51.853364 16049 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:51.853612 16049 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:51.854327 16049 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:32:51.855837 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:51.856072 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:32:51.856194 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:32:51.856426 16049 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:51.875666 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:51.947532 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:52.024187 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:52.052597 16049 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:52.052839 16049 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:52.053495 16049 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:52.055361 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:52.055544 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:52.055620 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:52.055895 16049 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:52.075484 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:52.128167 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:52.184718 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:52.227564 16015 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:52.227790 16015 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:32:52.228621 16015 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:52.230209 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:52.230394 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:52.230532 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:52.230839 16015 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:52.231714 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:52.253006 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:52.308591 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:52.414335 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:52.510977 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:52.569391 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:52.569756 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:52.682089 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:52.685168 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:52.696359 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:52.709627 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:52.790899 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:32:52.899061 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:52.950577 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:53.006500 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:53.108110 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:53.192438 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:53.253870 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.259326 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.372781 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:53.375666 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240502 21:32:53.407634 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:53.425379 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.426556 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:32:53.434955 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.453312 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:53.540175 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:53.619995 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.635195 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.668159 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.685691 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.751600 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:53.751678 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.763710 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.834133 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.847669 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:32:53.871596 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.883491 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.884236 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.916976 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.930497 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:53.942070 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.949641 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:53.956202 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 21:32:53.957276 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:53.986158 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.025488 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.048105 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.051733 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:54.092320 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:54.103003 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.132567 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.143541 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.201992 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:54.260002 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.284487 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.334440 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.337496 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:54.338038 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.402371 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.441884 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:54.453692 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.496443 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:54.534142 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:54.561300 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.599112 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:54.609210 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:32:54.733639 16049 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:32:54.733875 16049 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:32:54.734632 16049 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:32:54.736124 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:54.736286 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:32:54.736420 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:32:54.736654 16049 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:54.793509 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:54.859309 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:32:54.924809 16172 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:54.925019 16172 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:32:54.925702 16172 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:54.927456 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:54.927614 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:54.927759 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:54.928058 16172 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:54.936323 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:54.941466 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:55.093451 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:55.151866 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:55.223466 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:55.236790 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:55.333956 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:32:55.500129 16049 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:32:55.500373 16049 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:32:55.501225 16049 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:55.503266 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:55.503484 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:55.503595 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:32:55.503868 16049 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:55.530014 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:55.547621 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:55.613097 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:55.675652 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:55.680505 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:55.794617 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 21:32:55.796110 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:55.835215 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:55.977936 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.007756 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.018414 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.025864 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:56.074255 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:56.168787 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:56.181759 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:56.181780 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.205060 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:56.223227 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.230845 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.238746 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.245154 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.262548 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.273756 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.278019 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.407301 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.490559 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.506565 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:56.511847 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.539984 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:56.593374 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:56.618501 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.621520 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.622150 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.622838 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.678413 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.687978 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.734239 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:56.780396 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:56.789110 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.791561 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.799965 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.818725 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:56.990432 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:57.003465 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:57.008327 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:57.035893 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.083842 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:57.094897 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:32:57.104015 16250 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
W20240502 21:32:57.104349 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 21:32:57.104277 16250 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:32:57.105016 16250 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:32:57.106412 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:57.106567 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:32:57.106678 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:32:57.106889 16250 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:32:57.151252 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.194790 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.234668 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:57.435487 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.530442 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:57.559368 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:57.581951 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:57.643553 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.727576 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.821341 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:57.858189 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:57.875062 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 21:32:57.876338 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:32:57.883212 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:57.899399 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:32:58.106233 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:58.144158 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:58.239389 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:58.252285 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.293289 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.322013 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:58.324317 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:58.336694 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.419833 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.428458 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.446375 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:32:58.514583 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 21:32:58.540041 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.548170 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:58.652993 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.661422 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.678236 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.679837 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:58.716466 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.728744 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:58.778647 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:58.791188 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.868391 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:32:58.897922 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:58.916692 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.967705 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.967725 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:58.973697 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:32:58.974656 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.017339 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.122402 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:59.131664 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:59.136422 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.144946 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.147552 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.172103 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.175504 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.239786 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:59.357604 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.386920 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:59.393783 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.398288 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:59.401054 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.420838 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.447367 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.490371 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.495829 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.507742 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.538496 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.620855 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:59.652005 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:32:59.698137 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:32:59.709530 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:32:59.765856 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:32:59.812032 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:32:59.861665 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:32:59.884335 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:00.039458 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:00.189527 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:00.289413 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:00.300746 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:00.335484 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:00.483904 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:00.524948 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:00.551375 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:00.716836 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:00.728363 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:00.730094 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:00.842401 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:00.845227 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:00.867702 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:00.875283 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:00.896039 16250 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:00.896293 16250 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:00.897059 16250 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:00.899011 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:00.899216 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:00.899317 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:00.899585 16250 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:01.027292 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:01.086638 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.105664 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.111039 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.118209 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 21:33:01.136936 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.166425 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:01.185736 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.302560 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:01.326102 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.365762 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.422657 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.422664 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.432595 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:01.481318 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.483887 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:01.500602 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:01.590282 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.594830 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.624655 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.633598 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.657049 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.679364 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:01.709038 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.728945 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:01.763043 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.824805 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.854598 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.865630 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.884899 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.911485 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:01.968597 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:01.981621 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:02.001403 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:02.032873 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:02.067441 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:02.095252 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:02.104277 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:02.126461 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:02.136372 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:02.199615 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:02.225270 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:02.241216 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:02.364888 16348 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:02.365159 16348 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:02.365952 16348 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:02.367498 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:02.367697 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:02.367844 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:02.368126 16348 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:02.517498 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:02.571875 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:02.584977 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:02.603431 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:02.618621 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:02.762025 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:02.795879 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:03.034629 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:33:03.035665 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.054585 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:03.071871 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:03.080436 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:03.102641 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:03.170950 16250 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:33:03.171170 16250 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:33:03.171831 16250 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:33:03.173249 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:03.173542 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:03.173674 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:33:03.173928 16250 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:03.174896 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.187655 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.339740 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:03.448026 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:33:03.466651 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:03.493564 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.552752 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:03.557610 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:03.620090 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.623440 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:03.689344 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.740118 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.787086 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:03.812683 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 21:33:03.827787 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.866593 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.894897 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.904054 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.905083 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:03.922092 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:03.993212 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:33:04.015208 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:04.030524 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:04.037636 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.061367 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:33:04.062196 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.086844 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.165374 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:04.185915 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.201743 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:04.206029 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.247601 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.255800 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.261008 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.345095 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.362005 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:04.368796 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.393031 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.432600 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.470686 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:04.483894 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.535893 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:04.536799 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.565685 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.581764 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.596040 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:04.604897 16250 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:04.605190 16250 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:04.605973 16250 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:04.607760 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:04.607940 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:04.608036 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:04.608287 16250 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:04.638235 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:04.687633 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.699249 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:04.749424 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.770766 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:04.925705 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:05.052963 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:05.061709 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:05.081754 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:05.106675 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:05.119469 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:05.233552 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:05.542981 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:05.586758 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:05.605162 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:05.612900 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:05.634136 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:05.669767 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:05.734689 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:05.737542 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:05.873224 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:05.974388 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.024889 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:06.077068 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.162775 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:06.165144 16250 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:06.165345 16250 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:33:06.166020 16250 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:06.167383 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:06.167523 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:06.167631 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:33:06.167861 16250 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:06.198390 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:06.256266 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.277271 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:06.285938 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:06.299752 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.309998 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.354812 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.374008 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.398020 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.413499 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:06.456619 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 21:33:06.543721 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.577127 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:06.585830 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.597275 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.642966 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.643363 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.717098 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.742286 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:06.754449 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.768800 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.792491 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:06.833076 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.835973 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.840530 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:06.847796 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.892414 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.894507 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:06.948112 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:06.972481 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:06.998131 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:07.042898 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:07.052851 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:07.056243 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:07.108168 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:07.198091 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:07.285302 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:07.308331 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:07.390727 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:07.394810 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:07.427798 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:07.556313 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:07.586622 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:07.640349 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:07.709990 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:07.840821 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:07.852262 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:07.968751 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:08.032447 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.052654 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:08.126484 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:08.179493 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:33:08.203495 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.207602 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.246605 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:08.448514 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:33:08.449764 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:08.455516 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:08.519366 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:08.530329 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:08.551674 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.592152 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.662089 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:08.662299 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.682870 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.711889 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.755299 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.759253 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.873034 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:08.939205 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 21:33:08.979290 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.004524 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.010118 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.036739 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240502 21:33:09.059033 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.069023 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.079634 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.107420 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:33:09.112246 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.133858 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.140007 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:09.162693 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:09.180171 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.194598 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.199033 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:09.211630 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.221659 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.235946 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.307308 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.354012 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.395283 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.408727 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.498484 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.506290 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.508631 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.513692 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.546680 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.557859 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.593670 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.620383 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:09.651432 16778 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:09.651645 16778 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:09.652431 16778 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:09.653704 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:09.654611 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:09.654729 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:09.654999 16778 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:09.660429 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:09.819762 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.841518 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:09.846913 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:09.930459 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:10.000038 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:10.033360 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:10.083897 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:10.097514 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:10.232836 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:10.412632 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:10.542539 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:10.589309 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:10.630125 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:10.658174 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:10.696230 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:10.767549 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:10.800726 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:10.952337 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:10.991483 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.100900 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.147251 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.155678 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.185415 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:11.193511 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.205929 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.216164 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.245616 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.260114 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.312430 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.355201 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:11.374012 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.442027 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.457942 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.496327 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.507488 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.605794 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.605957 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 21:33:11.625191 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.627450 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.642789 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.670876 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.712322 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.721022 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.746486 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.747818 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.781493 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.801880 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:11.807466 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:11.834213 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.837939 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:11.899678 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.918144 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:11.949433 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:11.956112 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:12.017490 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:12.085672 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:12.187036 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:12.205840 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:12.303184 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:12.337114 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:12.340432 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:12.357254 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:12.393399 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:12.492902 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:12.532691 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:12.608318 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:12.785777 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:12.803089 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:12.830538 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:12.876879 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:12.955372 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:12.980227 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:12.983569 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:13.028133 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:13.193835 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:13.242844 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:33:13.418825 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:13.444655 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.461843 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.467495 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:33:13.475351 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:13.500399 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.547255 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:13.563156 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:13.660758 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.752087 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:13.787276 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.817332 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.832532 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.842934 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.890703 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:13.905506 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.920581 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.927096 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.935035 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.960254 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:13.960237 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:13.961716 24234 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.16.122.3:38427 timed out after 30.000s (SENT) [suppressed 1 similar messages]
W20240502 21:33:13.963879 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:13.995313 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 21:33:14.012030 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.023221 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:14.031309 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:14.067672 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240502 21:33:14.068770 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.069279 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:14.126264 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.135848 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:33:14.136763 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:14.162331 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.234877 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.238960 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.280405 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:14.281621 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:14.329216 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.406486 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.427820 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.446996 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:14.500631 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:14.552992 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:14.572840 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.578604 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:14.579869 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.621608 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:14.672698 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.814113 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:14.896414 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:14.903306 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:15.004395 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:15.024369 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:15.099941 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:15.148990 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:15.176792 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:15.185978 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:15.190850 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:15.233501 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:15.296644 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:15.355336 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:15.487946 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:15.514943 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:15.630036 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:15.729800 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:15.748293 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:15.879845 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:15.993647 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.049587 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.063066 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.067114 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 21:33:16.076702 17206 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:16.076936 17206 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:16.077690 17206 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:16.079275 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.079479 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:16.079632 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:16.079917 17206 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:16.098402 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.143309 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:16.145572 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.146154 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.151891 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.203770 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:16.216503 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:16.247485 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.260437 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.306911 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.349371 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.391528 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.417234 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.438530 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 21:33:16.451822 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.502111 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.589705 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.589913 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.622323 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.665637 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:16.688591 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.760515 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:16.772784 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.800202 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:16.806684 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:16.816150 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.821405 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.853221 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.908689 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:16.941546 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:16.945724 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:16.952962 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.075404 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.082902 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.107152 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.147385 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:17.172956 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:17.179860 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:17.210278 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:17.246595 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.389844 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.476652 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:17.507031 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.539567 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.566242 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:17.619027 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:17.638376 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:17.721402 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:17.722543 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:17.754386 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:17.811117 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.041138 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.055399 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.081180 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.129915 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:18.147572 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.153354 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:18.297329 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:18.302203 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:33:18.350868 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.442086 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.460678 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.497390 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.534231 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:18.543730 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:33:18.546685 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.576997 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.588070 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.589015 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:18.592082 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.602699 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.606056 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.673243 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.673419 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.696056 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.796075 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.837189 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.941291 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.948376 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:18.973609 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:18.978674 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:19.006009 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.036909 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:19.057561 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.077564 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 172 similar messages]
I20240502 21:33:19.141979 17406 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:19.142228 17406 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:19.143013 17406 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:19.144415 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:19.144608 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:19.144726 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:19.145017 17406 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:19.166597 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 21:33:19.201583 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:19.203464 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:19.213470 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.231007 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.243713 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:33:19.253473 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.299062 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.347880 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.350435 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:19.373701 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.488667 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:19.518996 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:19.567505 17373 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:33:19.567720 17373 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:33:19.568367 17373 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:33:19.569605 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:19.569864 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:19.570026 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:33:19.570312 17373 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:19.575378 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:19.595070 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:19.609022 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.632086 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:19.632527 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.775686 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.844411 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.855062 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.873350 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:19.898610 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:20.058830 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:20.088367 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:20.137838 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:20.168776 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:20.212842 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:20.329613 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:20.359484 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:20.411883 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.469731 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.523638 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:33:20.544122 17373 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:20.544428 17373 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:20.545239 17373 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:20.546972 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:20.547180 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:20.547282 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:20.547531 17373 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:20.554747 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:20.573940 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:20.607604 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:20.645051 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:20.821029 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.887028 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.923367 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.960263 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.960289 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:20.966526 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:21.051966 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.070106 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.082916 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:21.134469 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.157541 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:21.170553 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:21.185323 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:21.234599 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.264119 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.287551 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.350508 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.364339 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.375300 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.389286 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 21:33:21.501091 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.546123 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:21.550887 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:21.580255 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:21.593091 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:21.623150 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.709427 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.714582 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:21.719782 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.761584 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.765339 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.786300 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:21.842592 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:21.864777 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:22.003031 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:22.051157 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:22.054041 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.092641 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:22.128796 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:22.200261 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:22.202369 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.258215 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.270071 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.315325 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.418543 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:22.447576 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.511214 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:22.545842 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:22.545858 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:22.614701 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:22.629182 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:22.661190 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:22.743690 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:22.780805 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:22.891707 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:22.981400 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:23.020298 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.020399 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 21:33:23.101133 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:23.136396 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.190402 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.229159 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.256510 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.318068 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.356834 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.412065 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:33:23.413049 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.422613 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.425777 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:23.508055 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.545337 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:33:23.546327 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.546463 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.557221 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.581334 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.611315 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.628525 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.632536 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:23.662832 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.775678 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.827622 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 21:33:23.840842 17373 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:23.841077 17373 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:23.842017 17373 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:23.844254 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:23.844471 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:23.844588 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:23.844911 17373 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:23.891186 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:23.905771 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.914393 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.934229 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.946419 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:23.960693 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:23.997089 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.142290 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:24.144560 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:24.155604 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240502 21:33:24.159072 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.175483 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:33:24.178150 17495 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:24.178383 17495 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:24.179049 17495 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:24.180564 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:24.180740 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:24.180845 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:24.181072 17495 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:24.192483 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.223924 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:24.332715 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:24.337198 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.398778 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:33:24.502915 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.504083 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.510651 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.518767 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:24.546757 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:24.599541 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:24.628887 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.652720 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:24.711915 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:24.718860 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.741554 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.779469 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:24.812094 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:24.921634 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:25.115171 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:25.153244 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.178507 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:25.206892 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:25.216415 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:25.242647 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.393239 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:25.416785 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.445843 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.501116 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:25.566104 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:25.592487 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.593780 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:25.655790 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:25.697520 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.699728 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.724196 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.809195 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:25.852684 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.916968 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:25.965265 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:25.982933 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.000414 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:26.076475 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:26.091579 17373 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:26.091804 17373 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:33:26.092541 17373 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:26.094017 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:26.094208 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:26.094332 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:33:26.094631 17373 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:26.114660 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:26.141752 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:26.163599 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.182730 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.183674 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 21:33:26.195434 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.223883 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.308856 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.339360 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.417407 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:26.464686 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:26.496941 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:26.529295 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.558189 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.574450 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.577214 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:26.593940 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.610023 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:26.637575 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.710615 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.746662 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.795691 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:26.902423 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:26.910876 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:26.949322 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:26.969506 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.050854 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:27.090325 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:27.128784 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.131652 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 21:33:27.205731 17373 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:33:27.205953 17373 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:33:27.206821 17373 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:33:27.208297 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:27.208453 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:27.208575 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:33:27.208806 17814 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:27.270383 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:27.300040 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:27.300050 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:27.373354 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:27.393447 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.464937 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:27.482646 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.536617 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:27.598244 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:27.637359 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:27.678645 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.839879 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.864389 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:27.864668 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:27.898921 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:27.936336 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.011232 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:28.015111 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.037818 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.090018 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.090147 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:28.157959 17736 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:28.158192 17736 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:28.159106 17736 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:28.160440 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:28.160634 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:28.160733 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:28.160955 17736 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:28.182504 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.278129 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:28.346117 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.368743 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.385720 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:28.395236 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.431102 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:28.458302 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:28.488358 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:33:28.517998 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.561055 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.569433 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:28.573055 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 21:33:28.574121 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 21:33:28.601238 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.602047 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:28.620172 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.648243 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.816470 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:28.859751 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.862594 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.901973 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:28.916385 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:28.984257 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:28.998245 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:29.009908 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:29.100898 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.104565 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.117633 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:29.204952 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.204989 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.252817 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 21:33:29.287662 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.329427 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.376430 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:29.392035 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:29.443316 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:29.444538 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:29.561398 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:29.575829 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.614012 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:29.720283 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 21:33:29.758890 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:29.806277 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.861415 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:29.870909 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:29.877470 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.930874 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.934614 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:29.936627 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:29.981721 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.075462 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:30.259070 17871 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:30.259306 17871 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:30.260192 17871 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:30.262198 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.262483 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:30.262609 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:30.263034 17871 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:30.289469 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.316252 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.348027 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.367688 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.371043 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.409636 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:30.419266 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:30.437552 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.450441 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.470271 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:30.487592 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.625064 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:30.629755 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.643443 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.715801 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.779085 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.821619 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:30.843523 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.888387 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:30.888659 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:30.919970 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:30.924652 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:30.941395 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:31.018050 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.117795 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.159466 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.212751 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.259271 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.327512 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 21:33:31.347187 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:31.352762 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.375677 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:31.412564 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:31.437377 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:31.455436 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:31.473335 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.493443 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.607384 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.608211 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.640064 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:31.778124 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.783598 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:31.793181 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.888347 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.888797 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:31.891630 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:31.914475 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:31.956372 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:31.986943 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:32.109928 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.111429 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:32.188664 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:32.204501 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:32.266611 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:32.285985 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:32.290997 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:32.378818 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:32.381243 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.429795 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:32.467458 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:32.483729 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:32.589022 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:32.611984 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.683041 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.712004 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:32.806259 17814 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:32.806519 17814 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:32.807291 17814 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:32.808943 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:32.809165 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:32.809273 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:32.809557 17814 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:32.876499 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.884495 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:32.898850 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.928434 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:32.946151 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:33.006281 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:33.052209 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:33.120316 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.128270 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:33.249650 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.289683 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.291700 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:33.314291 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:33.346330 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.362957 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:33.470587 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:33.489756 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.512699 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.515358 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:33:33.516242 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.535280 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.604811 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:33:33.628042 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:33.726312 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:33.729269 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:33.810626 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.815605 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.829335 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.854998 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:33.861940 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:33.870867 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:33.917973 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.073904 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 21:33:34.104606 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:34.138995 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.151962 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:34.165386 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:34.247378 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:34.252099 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:34.258942 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 21:33:34.259841 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.270803 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.333587 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.422168 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:34.570381 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.588423 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:34.661597 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:34.680469 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:34.711763 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.724020 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.726681 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.752301 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.758486 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:34.760859 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:34.806694 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:33:34.850466 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:34.859894 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:34.937255 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:34.975535 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.017556 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.161890 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.241446 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.252542 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:35.255837 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:35.256072 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.258081 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:35.334306 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.337103 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.495189 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.541756 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.580901 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.631515 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.663635 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.764827 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:35.771827 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:35.800998 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.813797 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:35.839927 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:35.859946 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.962908 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:35.963833 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:36.012410 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.158818 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:36.159752 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.179198 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.259450 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.264554 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.266134 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.295605 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.310659 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:36.351987 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:36.378620 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.403592 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:36.417652 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:36.449154 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.473731 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.500607 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 21:33:36.591229 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:36.603801 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.625430 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.725814 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:36.793943 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:36.876948 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:36.917069 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.926753 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:36.933308 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:36.962255 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:37.063141 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:37.098325 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:37.188076 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:37.189620 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:37.257103 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:37.299932 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:37.314872 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:37.319785 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:37.365257 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:37.390296 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:37.483713 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:37.495049 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:37.525547 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:37.642930 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:37.723835 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:37.725368 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:37.733566 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:37.809970 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:37.821046 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:37.904614 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:38.008634 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:38.059847 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:38.082855 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:38.121462 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.138806 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:38.168223 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.190843 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.210744 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:38.310657 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:38.324970 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.336483 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:38.412243 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:38.419827 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.427476 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:38.522228 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:33:38.571799 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:38.651224 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:33:38.652266 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.709380 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.721416 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.723457 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:38.779451 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:38.829960 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.871387 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:38.872474 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:38.891942 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.936586 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 21:33:38.945449 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.952472 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.972184 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.976579 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:38.992486 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.044623 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.058430 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.136713 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:39.222726 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.229346 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.288779 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240502 21:33:39.309667 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.318583 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.356390 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.462306 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:39.471520 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:39.503839 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.522292 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.614856 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.666432 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.708586 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.741760 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.811281 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:33:39.832207 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:39.842629 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:39.846662 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.870381 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:39.929811 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:39.984476 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.030474 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:40.075729 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.100200 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.180099 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:40.245720 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.293311 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:40.308465 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.343976 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:40.369470 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:40.419380 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.432901 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:40.550238 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:40.567250 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:40.633574 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.684180 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:40.717764 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:40.792865 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.809443 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:40.843181 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:40.866384 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:40.879884 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.888219 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:40.967749 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:40.968572 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:41.048110 18390 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:41.048349 18390 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:41.049160 18390 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:41.051376 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:41.051584 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:41.051676 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:41.051903 18390 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:41.090863 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.150772 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.163765 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.277658 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:41.373669 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.408751 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.410566 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.411399 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:41.449533 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:41.459410 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:41.478132 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.491045 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 21:33:41.521515 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.545465 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:41.591396 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.601358 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.631633 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:41.656337 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:41.675820 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.768328 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.843837 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.905784 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:41.920917 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:41.923002 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:42.007972 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:42.064121 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:42.067749 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:42.070852 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:42.134773 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:42.191586 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:42.246670 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.320621 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:42.372859 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:42.425833 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.461062 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:42.483305 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.501531 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:42.519335 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:42.662421 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:42.677099 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.764874 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 21:33:42.782939 18390 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:42.783149 18390 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:33:42.783759 18390 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:42.785292 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:42.785476 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:42.785602 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:33:42.785858 18390 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:42.815470 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:42.844435 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.869498 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.874737 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:42.917898 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:43.020263 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:43.041568 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.053287 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:43.096908 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:43.123262 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:43.155568 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:43.221570 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.263746 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:43.316195 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:43.425493 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.459136 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.489336 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:43.560638 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.565848 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:33:43.595846 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.599510 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:43.606506 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:43.648465 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.659288 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.666533 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:33:43.676517 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:43.881281 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:43.884080 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:43.974097 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:44.006973 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.021564 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.045503 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.081089 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.107807 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.111106 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.170004 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:44.196264 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:44.202666 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:44.246585 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:44.271453 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.324038 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.350234 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:33:44.361301 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 21:33:44.390717 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.453061 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.501564 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:44.562526 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:44.577361 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.584292 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:44.672972 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:44.709398 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:44.709373 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.726836 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:44.757452 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:44.862588 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:33:44.866067 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:44.890064 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:44.991971 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:45.061226 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.099196 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:45.109726 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:45.279415 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:45.304504 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:45.331907 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.368268 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:45.378093 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.447320 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:45.461346 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:45.507702 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:45.508203 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:45.609514 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.611634 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.670181 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.694664 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:45.741132 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.760215 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:45.760353 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:45.776623 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:45.892287 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:45.917622 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:45.950479 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.019806 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.081008 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:46.087131 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.169780 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:46.171236 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.248009 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:46.282459 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.290907 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:46.310585 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.383211 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:46.531764 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.592088 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.592500 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.614629 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:46.698621 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.757040 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.768941 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:46.808418 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.854753 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:46.860931 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.868007 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:46.874891 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.910060 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.929720 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:46.960083 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:46.967784 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 21:33:47.011173 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:47.013957 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.031642 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:33:47.060073 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:47.071326 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 21:33:47.072512 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:33:47.156196 18390 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:33:47.156458 18390 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:33:47.157323 18390 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:33:47.159250 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:47.159438 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:33:47.159575 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:33:47.159845 18390 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:47.209232 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:47.243074 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:47.264276 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:47.282502 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.301106 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.389328 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.444690 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:47.460222 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:47.485735 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:47.500284 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:47.606714 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.694379 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:47.744793 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:47.815701 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:47.818506 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.831931 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:47.870441 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:47.887043 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:47.893296 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:47.904134 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.025514 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:48.037138 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.044342 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.120256 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.121968 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.213397 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.309085 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.315451 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:48.355618 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.380314 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:48.417557 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:48.477985 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.498716 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.513605 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:48.533479 18774 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:48.533790 18774 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:33:48.534615 18774 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:48.536123 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.536434 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:48.536587 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:48.536870 18774 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:48.568621 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.592566 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.671304 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:33:48.724232 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.758538 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 21:33:48.818059 18390 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:48.818310 18390 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:33:48.819177 18390 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:48.820595 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:48.820742 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:48.820840 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:33:48.821110 18390 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:48.868968 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.905354 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.909126 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.914706 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:48.930568 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:48.950024 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:48.970402 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:49.058650 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.104607 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.168054 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.227002 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.237197 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.312480 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.316828 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:49.353631 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 21:33:49.357185 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:33:49.358536 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:49.366760 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.442240 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:49.459062 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.464432 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:49.477720 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:49.504535 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:49.504961 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.531082 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.691696 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:49.722191 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:49.741524 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:49.755908 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:49.821728 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:49.861078 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.916004 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:49.925393 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:33:49.926403 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:49.944991 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:49.987783 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:50.032383 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:50.166286 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.253427 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:50.302850 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:50.364979 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:50.367035 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:50.375720 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.387159 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.392253 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:50.499943 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:50.502843 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.577903 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:50.662835 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.677181 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.732980 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.768525 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:50.813165 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:50.815106 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:50.817251 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:50.856161 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.876194 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:50.894208 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:50.921640 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:50.950656 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:51.098909 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.140197 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:51.157552 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.164189 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.217667 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:51.285384 18390 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:51.285614 18390 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:51.286496 18390 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:51.287791 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:51.288257 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:51.288425 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:51.288527 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:51.288810 18390 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:51.306514 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:51.373313 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.449402 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.466151 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:51.491933 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.544528 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.617926 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:51.638310 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.675582 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.731714 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:51.798542 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:51.811831 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:51.812485 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:51.821486 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.926208 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:51.941762 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 21:33:51.946568 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:51.956589 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:52.001816 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:52.007089 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.044579 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.102138 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:52.124866 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:52.154027 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.204571 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.269680 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:52.332943 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.346454 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:52.403566 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.404506 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:52.418927 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:52.519248 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:52.597110 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:52.617481 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:52.649111 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:52.801383 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:52.857614 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:52.932701 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:52.957914 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:53.005189 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.045861 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.064435 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:53.077800 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.082082 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.118992 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.203961 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:53.233729 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.329036 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:53.362365 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.383023 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.387634 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:53.408397 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:53.428179 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:53.436141 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.439363 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:53.567423 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.578203 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:53.651597 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:53.701272 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:33:53.766088 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:33:53.779237 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:53.796927 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.804172 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.860055 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.874125 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:53.913007 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.927752 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:53.929016 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.111584 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:54.121414 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.141608 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.154246 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.160213 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.254329 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.359318 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240502 21:33:54.397995 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.410732 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.413668 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.414579 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 21:33:54.494884 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:54.498502 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:54.514333 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:54.545970 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.611176 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.616711 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.630709 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:54.720561 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:54.731158 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.749017 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.830338 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:54.852311 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.873697 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:54.922264 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:54.929740 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:54.940541 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:33:54.991879 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:55.124754 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:55.281333 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:55.301760 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.331748 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.344856 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:55.375309 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.382761 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:55.389917 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:55.492486 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:55.560495 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:55.567870 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:55.597658 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.611454 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.660008 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.681248 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.826988 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:55.856671 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.876528 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:55.976004 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:55.976982 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:55.983251 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:55.993397 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:56.000568 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.005111 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:56.110373 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.139690 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:56.230672 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.275332 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.285269 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:56.306416 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.322296 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:56.373521 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:56.389823 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.427424 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.442639 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:56.462008 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.586454 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:56.744240 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:56.744994 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.752646 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.785565 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:56.863080 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.867508 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:56.898756 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 21:33:56.965246 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:56.996824 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:56.999102 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:57.007082 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.012928 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:57.043862 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:57.067390 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.070010 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:57.205219 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:57.225821 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:57.307379 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.316197 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.343319 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:57.576848 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:57.593310 19143 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
W20240502 21:33:57.593612 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:57.593530 19143 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:33:57.594224 19143 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:57.595654 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:57.596019 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:57.596133 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:33:57.596390 19143 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:57.654789 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:57.665906 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:57.666541 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:57.732776 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:57.755865 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:57.817536 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:57.831024 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.876816 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.894531 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:57.991206 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:58.026387 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.040746 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.063800 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:58.100888 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.122834 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:58.191416 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:58.223567 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.257436 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.273572 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.299257 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.361816 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:58.387359 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:58.445590 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:58.478593 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.523491 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:58.578625 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:58.624275 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:58.650530 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.661283 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.679611 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:58.680449 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.775732 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:58.780911 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:33:58.794412 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:58.902532 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:33:58.959810 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:33:58.966977 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:59.046869 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:59.054376 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.060125 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.081561 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.144352 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:33:59.164001 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:59.173811 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.222201 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.271629 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:33:59.304102 19143 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:33:59.304316 19143 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:33:59.305032 19143 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:33:59.306533 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:59.306744 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:33:59.306871 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:33:59.307152 19143 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:59.361960 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 21:33:59.402511 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 21:33:59.415086 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.434293 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.469821 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:59.481232 19143 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:33:59.481463 19143 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:33:59.482142 19143 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:33:59.483465 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:33:59.483649 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:33:59.483742 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:33:59.484063 19143 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:33:59.536240 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:33:59.538333 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.557978 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:33:59.563095 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:59.581326 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:33:59.704128 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:33:59.760871 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:59.774394 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:33:59.781652 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:33:59.953734 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:34:00.041435 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:00.055303 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:00.058010 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:00.098791 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.134975 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.169713 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.190595 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:00.268746 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.269197 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.317708 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:00.322957 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.336516 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:00.472388 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.518365 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:00.533067 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.560343 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:00.564803 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.565888 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.651651 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.658861 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:00.668711 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.696038 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.749189 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.752593 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.936337 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:00.938993 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.951782 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:00.979904 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:00.992415 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:01.039569 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.086884 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:01.159488 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:01.160349 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.177819 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.206460 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.354529 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:01.391266 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.424954 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:01.455248 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:01.488672 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.495816 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:01.596913 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.637135 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:01.697782 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.707629 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:01.764969 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 21:34:01.798560 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.827539 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:01.855659 19435 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:01.856611 19435 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:34:01.858071 19435 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:01.860044 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:01.860976 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:01.861222 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:01.861668 19435 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:01.873150 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:01.930239 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:01.957881 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:01.958101 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:01.986575 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.045537 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.049261 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:02.112828 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:02.144820 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.247007 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.286427 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.323755 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.395066 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:02.398464 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.439344 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:02.446966 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.522806 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:02.589483 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:02.622321 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.642872 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:02.697537 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.714232 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.742617 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.852170 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:02.866122 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:02.914296 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:02.917443 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:02.929100 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:02.954195 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.089468 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:03.096412 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.101790 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:03.106768 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.147684 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.176419 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:03.207403 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.285984 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.361675 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:03.365725 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:03.368054 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:03.399457 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.476419 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:03.628834 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.665360 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.679286 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.680898 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:03.728832 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:03.757305 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:03.762212 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.830646 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.871027 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:03.876405 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:03.920766 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:34:03.922582 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:34:04.017055 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:04.068359 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:04.075579 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:04.188175 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:04.198241 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:04.276168 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:04.293985 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:04.365566 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240502 21:34:04.395643 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:04.400703 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:04.476359 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:04.477468 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 21:34:04.548787 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:04.615687 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:04.642189 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:04.741838 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:04.749101 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:04.758430 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:04.767103 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:04.780223 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:04.785950 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:04.846706 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:04.889289 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:04.901590 19605 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:04.901798 19605 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:34:04.902563 19605 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:04.904119 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:04.904325 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:04.904440 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:34:04.904660 19605 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:04.972455 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:04.997120 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:34:04.998034 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:05.081374 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.101917 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:05.121783 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:05.185101 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:05.209718 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:05.212561 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:05.233135 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:05.297530 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:05.313997 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:05.373549 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.400072 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.403872 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:05.423813 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.446048 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.472820 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.568140 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:05.669175 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.676362 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:05.721587 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.735293 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:05.770094 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.806902 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:05.856768 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:05.885138 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:05.894116 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.972775 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:05.988147 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:06.015676 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:06.050437 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:06.064666 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:06.115953 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:06.179569 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:06.310408 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:06.339150 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:06.379119 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:06.405858 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:06.467034 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 21:34:06.480913 19605 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:34:06.481169 19605 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:34:06.481832 19605 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:34:06.483240 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:06.483425 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:34:06.483538 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:34:06.483753 19605 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:06.642599 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:06.662853 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:06.718216 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:06.732318 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:06.805260 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:06.872283 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:06.907310 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:06.947803 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:06.987896 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.014158 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.015203 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.129484 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 21:34:07.163416 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.179562 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:07.239342 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.246713 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:07.307776 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.329236 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.361850 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.444483 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.474224 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.477195 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.568668 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.572562 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.592403 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.619943 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.648912 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.674225 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:07.677240 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:07.704396 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.710156 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.716063 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.819772 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.825361 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:07.841090 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:07.852717 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:07.938264 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.091958 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:08.110147 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:08.112963 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:08.222543 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.275585 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:08.303529 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:08.320756 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.347366 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.360198 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.393090 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:08.403185 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.446399 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.463872 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.576450 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:08.582922 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:08.633199 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:08.638847 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.696861 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.735352 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:08.773376 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:08.792238 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.792964 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:08.958061 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:34:08.959317 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:08.970549 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:34:09.053882 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:09.144830 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:09.170177 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:09.174016 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:09.254657 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:09.328014 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:09.398026 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:34:09.488894 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:09.492950 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.494020 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 21:34:09.549618 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:09.574947 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.604341 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:09.612083 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:09.672750 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:09.675681 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:09.801838 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.850996 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.896135 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.922797 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.924497 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:09.926607 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:09.932943 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:09.953894 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.962292 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:09.963043 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:09.985447 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.071244 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.071599 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:10.127007 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:10.136050 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.151693 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.169659 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:34:10.170650 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.174747 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:10.225556 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.343490 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.348562 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:10.409874 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:10.422814 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.466653 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.483186 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.540170 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.547869 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:10.653630 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:10.798727 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.813727 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.860541 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:10.874900 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.875401 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.879894 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.885859 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.888738 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:10.912431 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:10.924103 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:10.993062 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:11.142179 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:11.180125 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:11.270752 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:11.275887 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:11.342027 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:11.401710 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:11.417593 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:11.496677 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:11.504384 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:11.667280 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:11.674667 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:11.872581 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:11.894336 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:11.911170 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.005775 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.023269 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.049749 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.076275 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:12.139801 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.193360 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.227340 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:12.289582 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.314940 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.344336 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.348071 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.355407 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 21:34:12.368734 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.379587 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.396783 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.411342 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.425901 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.517096 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.529635 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.608158 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.611282 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:12.638322 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 21:34:12.669981 19605 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:12.670202 19605 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:34:12.671087 19605 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:12.671875 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:12.672236 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:12.672336 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:34:12.672547 19605 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:12.720687 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.780045 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.785032 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:12.813728 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.818285 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:12.896690 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.899536 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:12.906947 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.911000 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:12.928606 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:13.039878 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:13.082391 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:13.198923 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.205276 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:13.319732 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.333051 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:13.358136 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:13.377086 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:13.471405 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:13.549237 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.561873 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.563894 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.581423 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:13.584463 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:13.616693 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.730448 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.739353 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.740142 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.748664 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:13.810271 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:13.888455 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:13.980314 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:34:13.981304 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:14.100847 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:14.104905 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:34:14.178889 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:14.228572 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:14.307121 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:14.381889 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:14.445710 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:34:14.459921 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:14.493160 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:14.523424 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:14.552804 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:14.554225 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:14.582413 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:14.600355 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:14.667268 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:14.761484 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:14.782223 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:14.851656 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:14.900007 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:14.947492 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:14.953675 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 21:34:14.969528 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.005472 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.010888 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.016875 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.022737 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.037900 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.045483 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.076277 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.093550 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.110217 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:15.176259 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:34:15.255993 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.265996 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.302716 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:15.324388 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.388011 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.415859 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.457501 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.534816 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.570403 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.620203 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.647787 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:15.654781 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.665105 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.778977 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:15.831226 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.888409 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.919668 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:15.919747 20157 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:15.919936 20157 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:34:15.920660 20157 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:15.921897 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.922582 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:15.922695 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:15.922930 20157 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:15.954772 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.982947 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:15.985885 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:15.988785 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:16.033339 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:16.106510 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:16.162709 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:16.163502 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:16.171059 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:16.283892 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:16.306819 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:16.436903 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:16.444900 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:16.486281 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:16.514693 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:16.577706 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:16.833698 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:16.847332 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:16.901577 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:16.987102 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.034940 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:17.050225 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.050746 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.060508 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.064327 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:17.065613 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.136893 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:17.139276 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.283555 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.343818 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.350203 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:17.353130 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 21:34:17.371528 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.413794 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.456614 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.482131 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.484376 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:17.530640 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.560827 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.576567 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:17.588867 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:17.603560 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:17.616547 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.676383 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:17.753238 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:17.757182 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:17.769485 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 21:34:17.827167 19605 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:17.827409 19605 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:34:17.828119 19605 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:17.830016 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:17.830229 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:17.830377 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:17.830689 19605 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:17.980831 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:18.008152 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:18.010864 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:18.062947 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:18.069463 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.132673 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.177991 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.188022 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.194770 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:18.237150 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:18.262611 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.264504 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:18.269402 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.465075 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.469650 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:18.515507 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:18.565194 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:18.568233 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.664448 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:18.680244 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.733536 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.750970 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:18.773448 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.805990 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:18.836966 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:18.975666 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.010118 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:34:19.030339 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:19.050392 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:19.106650 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:34:19.250147 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.254384 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:19.271229 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.421911 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.496865 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:19.558547 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240502 21:34:19.582269 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:19.586639 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:19.600889 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.619014 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.636991 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.642490 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:19.665596 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.692893 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:19.758337 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.782954 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.816121 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.834988 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.845438 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:19.848346 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:34:19.875516 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:19.885972 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.896512 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:19.909621 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:19.939301 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 21:34:20.052704 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.078845 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:20.101727 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:20.169157 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:20.250794 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.270129 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:20.345270 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:34:20.346388 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.355029 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.377485 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:20.434201 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:20.447037 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:20.497200 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.540614 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:20.562126 20380 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:34:20.562376 20380 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:34:20.563071 20380 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:34:20.564741 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:20.564903 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:20.565053 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:34:20.565330 20380 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:20.578218 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.589408 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:20.624927 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:20.627084 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:20.674432 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.690394 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.778625 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.855595 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.863870 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:20.939317 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:20.977783 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.014650 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:21.035176 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:21.036598 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:21.042008 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.145195 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:21.163286 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:21.322053 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:21.349248 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:21.359683 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.371924 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.401173 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:21.646489 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:21.651072 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.716388 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:21.843547 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.909034 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:21.933291 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:21.988721 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.035976 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:22.092446 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:22.118324 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.136854 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:22.232419 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.240887 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.307091 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.323236 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.343291 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:22.377514 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.399966 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:22.411268 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.412778 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.451699 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.493273 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.498472 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 21:34:22.517550 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.518594 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:22.579257 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.581982 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:22.645380 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.654058 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:22.714774 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.775120 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:22.812333 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.844450 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:22.914741 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:22.955515 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:23.034190 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:23.044323 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.061481 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:23.062078 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.069856 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:23.089735 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:23.206207 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:23.253722 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:23.258003 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.273460 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.295558 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.299877 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.311399 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:23.380312 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.419974 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:23.554673 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:23.557323 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.601548 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:23.612450 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.618691 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:23.656276 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.700929 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:23.833240 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:23.851449 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:23.969658 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:24.044479 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:34:24.050225 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:24.092218 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:24.110577 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:34:24.206787 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:24.378177 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:24.480628 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:24.515739 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:24.516564 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:24.521142 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:24.606750 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:34:24.624991 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:24.638629 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:24.683312 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:24.714663 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:24.743889 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:24.765118 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 21:34:24.783252 20469 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:24.783463 20469 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:34:24.784085 20469 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:24.785626 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:24.786465 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:24.786590 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:34:24.786868 20469 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:24.801959 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:24.808635 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:24.900889 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:24.903905 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:24.957911 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:24.985535 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.000146 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:25.000145 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.037539 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.044888 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.054997 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.060437 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.064983 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 21:34:25.090972 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.093109 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.147117 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.218175 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:25.379964 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.396194 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.421765 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:34:25.422690 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.432238 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.436230 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:25.438586 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.440191 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.527046 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.548177 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.596657 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.605834 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.718945 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.724610 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.751436 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:25.774012 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.813057 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:25.837698 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:25.858009 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.870817 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:25.911866 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:25.970577 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.048202 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:26.196431 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:26.200438 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:26.216609 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:26.234694 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:26.318734 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:26.350200 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.394201 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:26.422549 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.457116 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.773491 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:26.881981 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.913532 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:26.934971 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.975327 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:26.978179 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:27.046767 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.059470 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.086298 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.228822 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.235888 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.242321 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:27.291456 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.317760 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.327569 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.344313 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.396366 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:27.406271 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.426505 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:27.473517 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.475204 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:27.501856 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:27.586155 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.635128 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.649271 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.689013 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 21:34:27.782756 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.791389 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:27.835256 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:27.849414 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:27.850593 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.851613 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.928815 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.943339 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:27.989399 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:28.007848 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:28.029750 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:28.069337 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.080855 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.090157 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:28.152529 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.218119 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:28.280100 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:28.319064 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.329830 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.415050 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:28.443605 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:28.467235 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:28.540572 20753 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:28.540771 20753 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:34:28.541517 20753 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:28.543063 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:28.543251 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:28.543340 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:28.543573 20753 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:28.608023 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:28.658255 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.677703 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:28.709612 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.733834 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:28.802618 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.815212 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:28.824173 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.837700 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:28.945770 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:29.017036 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:29.074146 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:34:29.113354 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 21:34:29.157550 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:29.195091 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:29.347693 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:29.365298 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:29.380400 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:29.457922 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:29.464208 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:29.547804 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:29.556161 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:29.593657 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:29.668483 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:34:29.696116 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:29.699103 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:29.735023 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:29.736423 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:29.736835 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:29.821591 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:29.907299 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:29.980882 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:30.018064 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.027158 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.029912 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.040910 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.041452 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:30.054512 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.146100 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:30.163267 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:30.206681 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.218077 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.222457 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 21:34:30.228418 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.246333 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.264847 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.333935 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:30.395823 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.443392 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:30.543705 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:30.567327 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.587590 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.650521 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 21:34:30.674787 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.676903 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.680932 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:30.758656 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:30.770149 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.837800 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:30.840869 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:30.852793 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:30.922767 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:30.973389 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:31.019845 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.067533 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.105763 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.158810 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:31.184672 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:31.358280 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:31.372087 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.403646 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:31.404036 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:31.408776 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.530979 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.533636 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:31.682211 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.709723 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:31.781782 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:31.852707 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:31.877033 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:31.893821 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:31.921787 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:32.166455 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:32.228439 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.252084 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.252087 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.262400 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.267033 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:32.291630 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:32.305642 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.364432 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:32.374164 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:32.375802 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.377082 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.473165 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.487247 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.539434 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 21:34:32.544615 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.570051 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.626591 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.644479 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.649103 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.697300 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:32.737296 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:32.745304 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:32.799927 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:32.825945 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:32.836472 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:32.860564 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.032236 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.044350 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.060686 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.065297 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:33.103847 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.199682 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.237098 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:33.268692 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:33.271904 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.372067 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.421439 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.429395 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.429771 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:33.469866 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.528229 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:33.574054 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.733321 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:33.746809 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.789690 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:33.792616 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.805027 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:33.846487 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.859071 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.904219 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:33.948714 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:33.992681 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:34.018697 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:34.079706 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:34:34.124439 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:34:34.325680 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:34.400260 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:34.409746 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:34.460402 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:34.501266 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:34.549518 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:34.558387 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:34.573041 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:34.620133 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:34.620584 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:34.693758 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:34.702112 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240502 21:34:34.884874 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:34.901084 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:34.962574 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:34.987792 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.033674 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.040127 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.050562 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:35.109948 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.123338 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:35.142680 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 21:34:35.166846 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:35.175969 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.216888 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.239382 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:35.267380 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:35.284885 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 21:34:35.321102 20822 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:35.321287 20822 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:34:35.321940 20822 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:35.323427 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.323601 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:35.323712 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:35.323976 20822 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:35.335628 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.349300 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:35.360893 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.380501 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.495327 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.508886 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.517192 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:35.536865 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:35.538444 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.597913 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.660779 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.666633 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:35.707594 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.732091 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:35.780973 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:35.800418 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:34:35.901335 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:35.974543 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.005934 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.007097 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:36.014621 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.064473 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:36.114957 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:36.136835 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:36.148094 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.184265 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:36.296397 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:36.332127 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.393502 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:36.593653 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:36.599444 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.621843 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:36.625583 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:36.629868 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:36.660897 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:36.741528 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:36.868137 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:36.963215 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:36.999680 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.028218 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.040225 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:37.049535 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.111620 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.178303 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:37.180552 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.272436 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.399032 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.425199 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.428767 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:34:37.472244 20753 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:37.472466 20753 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:34:37.473191 20753 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:37.474617 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:37.474854 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:37.475018 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:34:37.475276 20753 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:37.487672 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.502624 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.551373 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.564316 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.566030 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:37.574062 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:37.577585 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.619475 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.667018 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.702055 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:37.724071 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.768895 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.794692 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.839403 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.849857 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:37.910933 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 21:34:37.941879 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:37.999984 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.054111 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:38.082309 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:38.101178 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:34:38.118387 21240 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:38.118633 21240 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:34:38.119572 21240 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:38.121722 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.123054 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:38.123270 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:38.123538 21240 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:38.126159 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:38.146070 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:38.184923 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:34:38.202586 20753 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:34:38.202838 20753 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:34:38.203646 20753 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:34:38.205349 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:38.205796 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:38.205929 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:34:38.206172 20753 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:38.302624 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.314121 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.325291 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:38.332315 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.398128 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.438524 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.462023 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.468117 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.547957 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.550558 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:38.593907 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.676987 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.691104 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:38.731711 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:38.838948 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.865633 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:38.934808 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:38.937276 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:39.051944 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.124330 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:34:39.124701 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:34:39.142616 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.164752 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:39.184765 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.213239 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:39.237957 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:39.356739 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:39.376890 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:39.402099 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:39.435827 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.437137 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.530510 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:39.563391 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:39.640868 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:39.705993 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 21:34:39.721136 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:39.743582 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:39.857476 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.909241 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:39.952311 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:39.998597 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.002645 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:40.052018 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.065109 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.071374 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:40.086900 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:40.123699 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.158788 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:40.172842 20753 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:34:40.173064 20753 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:34:40.173844 20753 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:34:40.175657 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:40.175839 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:34:40.175959 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:34:40.176304 20753 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:40.210394 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:40.217175 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 21:34:40.284356 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:40.325379 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.373435 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.449589 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.464118 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:40.498643 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:40.536857 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:40.587667 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:40.588626 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:40.645134 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:40.652432 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.656448 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:40.673857 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.686820 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.704773 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:40.860649 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:34:40.863058 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.869527 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:40.918843 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.962913 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:40.984709 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:41.012198 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:41.050776 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.064139 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.083361 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:41.116302 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:41.256696 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.328912 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:41.453207 24237 maintenance_manager.cc:419] P 34199807c207492781e8dba15a3f67ab: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033391
W20240502 21:34:41.455806 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 21:34:41.460353 24236 maintenance_manager.cc:643] P 34199807c207492781e8dba15a3f67ab: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.002s Metrics: {"bytes_written":1650,"delete_count":0,"lbm_write_time_us":245,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":236,"threads_started":1,"update_count":15,"wal-append.queue_time_us":321}
I20240502 21:34:41.461082 24237 maintenance_manager.cc:419] P 34199807c207492781e8dba15a3f67ab: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.114218
I20240502 21:34:41.475543 24236 maintenance_manager.cc:643] P 34199807c207492781e8dba15a3f67ab: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14231,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":626,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":136,"lbm_read_time_us":295,"lbm_reads_lt_1ms":17,"lbm_write_time_us":309,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":443,"threads_started":2,"update_count":15}
W20240502 21:34:41.493049 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:41.519205 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:41.525565 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.563405 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.740481 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.771343 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:41.775753 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:41.776408 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 21:34:41.850633 20753 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:34:41.850889 20753 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:34:41.851583 20753 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:34:41.853132 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:41.853745 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:41.853881 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:34:41.854120 20753 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:41.910213 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:41.916908 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:41.949329 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:41.955744 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.003165 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:42.006255 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.105849 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.124336 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.146733 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:42.236624 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:42.365582 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.399567 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:42.435396 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.441156 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.453310 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.458994 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.472153 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:42.489214 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:42.493360 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.518206 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.532200 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.625907 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.672734 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:42.710832 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 21:34:42.732273 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:42.814965 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:42.833088 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.876199 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.905102 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:42.934056 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:42.984274 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:43.086966 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.088832 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:43.135674 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:43.142766 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:43.232297 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:43.251473 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:43.302340 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:43.308401 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.308429 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.319598 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.365507 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:43.381027 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.383294 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:43.544430 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.574612 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.635830 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:43.668684 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:43.710304 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:43.741873 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:43.819458 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:43.870548 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:43.951434 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:44.089488 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:44.093091 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:44.106287 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:44.148901 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:34:44.155982 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:34:44.214241 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:44.241258 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.247026 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:44.249171 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.379954 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.416419 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:44.425905 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.480082 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:44.499619 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:44.506088 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:44.509672 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.607897 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:44.614540 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.654518 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:44.674585 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.726248 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:34:44.823556 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:44.858839 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:44.881716 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:44.925048 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:44.980962 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.047407 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:45.070566 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:45.113966 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.127619 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:45.199898 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:45.216221 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:45.223160 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:45.248281 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.260644 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:45.289678 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 21:34:45.343744 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.347690 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.349893 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.395617 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.403131 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.409386 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:45.520131 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.543183 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:45.621591 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:45.653087 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:45.743422 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:45.744988 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:45.764195 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:45.853927 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:45.875702 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:45.915781 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:34:45.919576 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:46.005867 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.040328 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.050850 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.072778 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:46.078994 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:46.091279 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.140493 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:46.151261 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.158150 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:46.225654 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:46.349998 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:46.387856 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 21:34:46.395442 21616 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:46.395706 21616 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:34:46.396528 21616 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:46.398051 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:46.398319 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:46.398465 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:46.398736 21616 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:46.465553 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.527976 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:46.615633 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:46.658072 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:46.706120 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.748206 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:46.766543 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:46.866163 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:46.866946 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.989611 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.992457 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:46.995946 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:46.999990 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:47.006242 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:47.012470 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:47.042695 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:47.105916 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 21:34:47.107124 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:47.136356 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:47.239702 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:47.295388 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:47.455962 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:47.475730 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.480729 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.491071 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:47.515548 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:47.537915 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.607594 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.617180 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:47.713439 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:47.735337 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.794865 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.811707 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.847966 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:47.862098 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:47.905189 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:47.916591 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 21:34:47.936256 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.956724 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:47.995872 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.020270 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.091135 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:48.099701 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:48.181946 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:48.196092 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:48.268035 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.282720 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:48.321661 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.388522 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.396476 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.407505 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:48.491796 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:48.506394 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:48.516557 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.601272 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.621330 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.702479 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.769529 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.797653 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:48.887543 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.889155 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:48.928818 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:48.963462 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:48.995649 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:49.009860 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.122488 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.168932 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:34:49.254732 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:49.263268 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:49.267443 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:49.326386 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:34:49.363238 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:49.380015 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.380703 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:49.411724 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:49.426724 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:49.497355 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.506140 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.582507 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.601651 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.651553 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:49.677130 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:49.729974 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:49.756531 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240502 21:34:49.781306 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:49.804563 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:49.819347 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:49.859601 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:49.862665 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:49.921662 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:49.946866 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:49.978049 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:50.058939 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.129071 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.151540 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.303262 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.326025 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:50.327476 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:50.340684 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:50.371882 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:50.373925 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.382740 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:50.387388 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:50.426241 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 21:34:50.440412 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.491088 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:50.504592 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:50.704629 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.721920 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:50.781958 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.809063 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:50.817612 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:50.820904 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:50.905416 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:50.913586 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:50.955701 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.974565 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:50.980865 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:34:50.981266 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:51.062189 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:51.084883 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.126823 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:51.192719 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:51.257481 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:51.263733 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:51.305429 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:51.377254 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:51.379336 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.398171 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:51.453709 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.547014 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.601414 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.613248 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:51.655702 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:51.806789 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:51.822081 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.905719 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:51.954727 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:51.991892 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.008857 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.009120 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:52.017716 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.052393 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.064263 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.183495 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:52.187232 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.199671 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:52.270272 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.326680 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.419833 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:52.447403 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:52.464049 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:52.531855 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.639958 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:52.643373 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.650130 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.720115 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:52.755009 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.776976 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.786463 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:52.802927 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.842240 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.946112 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.950161 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:52.961738 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:52.972180 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.025575 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:53.062919 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:53.080307 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:53.089828 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:53.142141 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 21:34:53.258533 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:53.433120 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.445205 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.467617 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:53.482385 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:53.548547 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:53.554773 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.597718 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 21:34:53.605446 21990 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:53.605701 21990 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:34:53.606559 21990 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:53.607928 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:53.608350 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:53.608490 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:34:53.608762 21990 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:53.624859 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:53.632215 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.682129 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:53.697816 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:53.735860 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:53.743436 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:53.745051 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.860592 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:53.935586 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:53.969724 21990 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:53.969985 21990 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
W20240502 21:34:53.970172 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:53.970786 21990 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:53.972473 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:34:53.972698 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:53.972882 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:53.973199 21990 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:53.975869 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 21:34:53.985975 21869 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:53.986208 21869 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:34:53.986959 21869 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:53.988873 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:53.989049 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:53.989197 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:34:53.989531 21869 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:54.058602 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.086879 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:54.094446 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:54.206605 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:34:54.207748 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.208516 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:54.244148 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:54.312266 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:54.356730 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.399320 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:34:54.471246 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:54.479394 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:54.488299 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.592566 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:54.650305 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:54.664891 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.678754 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:54.767614 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:34:54.768582 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.798735 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:54.870209 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:54.915865 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:54.950122 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:54.980289 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.019752 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:55.162163 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.184340 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.212093 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.230057 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.230726 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:55.230852 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.243091 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:55.264892 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.293668 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.373945 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:55.438750 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.445614 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.472308 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.494894 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:55.505890 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:55.522501 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.556893 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 21:34:55.591257 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.645532 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:55.843950 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:55.903264 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:55.948848 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:55.962579 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:55.997653 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:56.017968 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:34:56.084425 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:56.090912 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.153028 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:56.188793 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.224627 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:56.224929 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.245193 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.248874 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.360484 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:56.368907 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:56.399165 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:56.453063 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:56.459105 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:56.463150 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:56.576860 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.578233 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.705612 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.773555 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:56.777715 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:56.794806 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:56.874333 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:56.874368 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:56.881359 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:56.945708 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:56.949805 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:57.010485 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:57.100775 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.171269 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:57.280603 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.294437 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:57.295917 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.387697 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.403084 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.506480 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:57.515323 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:57.521054 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.571508 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:57.637614 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.648785 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.661849 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.669131 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.731228 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:57.810572 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.881829 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:57.909344 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:57.915740 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.928126 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:57.929051 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:57.988451 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 21:34:58.017674 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:58.060557 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:58.126216 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:34:58.141235 22054 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:34:58.141464 22054 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:34:58.142243 22054 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:34:58.143604 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:58.143910 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:34:58.144023 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:34:58.144325 22054 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:58.221662 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.285387 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:58.303208 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.327085 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:58.339164 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:58.479681 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.521807 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:58.534457 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.605101 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:58.624701 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.685148 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:58.704427 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.717572 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:58.749558 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:58.765617 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.795312 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:34:58.839751 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.840243 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:58.927194 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:59.056437 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:59.093861 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:59.119825 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:59.175289 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:34:59.192543 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:34:59.229858 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:59.243117 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:59.261826 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:34:59.262893 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:59.291399 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:34:59.399529 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:34:59.400007 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:34:59.402822 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:59.446208 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:34:59.485611 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:59.568195 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:59.615276 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:34:59.637522 22054 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:34:59.637739 22054 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:34:59.638589 22054 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:34:59.640087 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:59.640280 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:34:59.640435 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:34:59.640676 22054 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:34:59.726702 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:34:59.789117 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 164 similar messages]
W20240502 21:34:59.811405 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:59.837239 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:34:59.853482 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:34:59.959733 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:00.023201 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.030737 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.032166 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:00.042140 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.053740 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.071795 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.135522 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.156909 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:00.169466 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:00.170917 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:00.176429 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.342334 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.364173 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 21:35:00.428009 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.478370 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:00.503253 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.508949 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:00.584108 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.669901 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.700522 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.764513 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:00.786420 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:00.855010 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:00.923383 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.928359 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.961262 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:00.976732 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:00.988147 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:00.991606 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.078709 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:35:01.136201 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.162236 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.179111 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:01.255334 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.269163 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:01.322485 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.352095 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:01.395977 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:01.440047 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:01.441133 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.479338 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:01.482200 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:01.501322 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.638824 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.684136 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:01.693910 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:01.729739 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:01.746488 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:01.779086 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:01.816831 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:01.897853 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:01.919363 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.002669 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.020430 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.094585 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:02.095510 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.205734 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:02.219774 22412 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:35:02.220000 22412 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:35:02.220723 22412 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:35:02.222227 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:02.222452 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:35:02.222610 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:35:02.222872 22412 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:02.228927 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:02.277681 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.325592 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:02.337863 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.375236 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.420601 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:02.446846 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.471045 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.489624 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:02.570924 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.624634 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.641805 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.695909 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.733817 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:02.760641 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 21:35:02.791749 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.798887 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:02.821048 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:02.886649 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:02.895833 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:02.910709 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:02.956874 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:03.086930 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:03.113531 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:03.169310 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:03.228119 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.284626 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:03.309851 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.329294 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:03.393326 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:03.480409 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:03.559434 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.568305 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.603281 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:03.711000 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.715975 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:03.733812 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.754513 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:03.762396 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.794037 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:03.869208 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:03.871511 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.881212 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:03.924366 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:03.990640 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:03.998248 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:04.070154 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.091912 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.100841 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.102138 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:04.127420 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:04.255196 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:04.263326 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:35:04.278079 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.297439 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:04.316190 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:04.362514 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:04.398555 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:04.415174 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:35:04.530447 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:04.586216 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:04.646483 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:04.691915 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:04.693239 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.713889 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:04.768788 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:04.778033 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:04.782517 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.791018 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240502 21:35:04.851500 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:04.941573 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:04.976969 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:05.108351 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:05.117102 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:05.117159 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 21:35:05.119165 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:05.158949 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:05.165745 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:05.172048 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:05.179975 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:05.198212 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:05.240666 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:05.247632 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:05.282868 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:05.494618 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:05.546595 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:05.583225 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:05.624287 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:05.629063 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:05.635599 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:05.687170 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:05.689322 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:05.741531 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:05.875517 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:05.926503 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:06.030093 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:06.098098 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:06.106936 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.110693 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:06.114015 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:06.131134 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:35:06.140991 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.223898 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:06.227523 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.248426 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.279819 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.355831 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.356086 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:06.394405 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.486197 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.497156 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.542147 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:06.578725 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:06.579455 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:06.593883 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:06.617460 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:06.625948 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:06.665472 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:06.682736 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:06.859704 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.016294 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.026090 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:07.060905 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:07.073487 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.079183 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.097486 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.135596 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.144938 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.150178 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.224051 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.357268 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.382426 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.401695 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:07.438237 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.510526 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.530676 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.564550 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.564555 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.591290 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:07.617440 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 21:35:07.623565 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.716470 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.810452 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.822338 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.836727 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:07.909207 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:07.924140 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.968788 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:07.986114 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:07.999797 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:08.044708 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.109289 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:08.217979 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:08.344825 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.382988 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:08.424022 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:08.465358 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:08.482982 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.500160 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:08.557277 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.558094 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.674432 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:08.681164 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.709903 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:08.728044 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:08.739423 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.759274 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.866475 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:08.942739 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:08.988782 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.002879 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.004518 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:09.098918 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:09.152230 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:09.152343 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.165560 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:09.170037 22633 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:09.170207 22633 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:09.170812 22633 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:09.172097 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:09.172252 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:09.172376 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:09.172601 22633 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:09.256381 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:09.281257 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:09.327338 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:35:09.382037 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.398018 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.415375 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:09.422765 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:09.437122 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:09.463975 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:35:09.474967 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 21:35:09.489320 22731 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:09.489537 22731 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:09.490221 22731 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:09.491787 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:09.491945 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:09.492069 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:09.492311 22731 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:09.528578 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.535200 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.552536 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.630753 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.699579 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.730322 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:09.811290 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:35:09.812327 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.913282 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.936302 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:09.946885 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:09.954198 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:09.994638 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:10.022511 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:10.048195 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:10.132189 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:10.143997 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:10.184159 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:10.198979 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:10.231576 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 21:35:10.338313 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:10.408988 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:10.427242 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:10.440135 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:10.483417 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:10.519181 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:10.560026 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:10.674018 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:10.715658 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:10.792167 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:10.805227 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:10.823719 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:10.903482 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:10.954337 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:10.959888 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.046921 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.114631 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.142282 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:11.179705 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:35:11.204150 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:11.204169 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:11.215989 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.304574 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:11.330777 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.330946 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:11.350534 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.353927 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.388159 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:11.475463 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.562713 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:11.688738 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:11.783952 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.792939 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:11.820281 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.847258 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:11.859984 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:11.886678 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:11.911674 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:11.938992 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:11.967270 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.053725 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.081387 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.106765 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:12.136719 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.163089 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:12.175274 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.180938 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:12.232173 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.327759 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.348886 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:12.391813 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.418895 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.424945 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.466929 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:12.516929 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.597816 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 21:35:12.600304 22731 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:12.600538 22731 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:12.601222 22731 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:12.602733 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:12.602927 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:12.603065 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:12.603369 22731 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:12.625684 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.634703 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:12.657163 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:12.669458 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.741003 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:12.833276 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.861974 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:12.930143 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:12.947813 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:12.954525 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:12.961767 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 21:35:13.117209 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:13.131474 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:13.139705 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:13.215574 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:13.255319 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:13.310660 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:13.379401 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.537194 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:13.566133 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:13.600446 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:13.601245 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:13.606534 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.639881 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.702420 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.735195 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:13.833439 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.847379 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.924579 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:13.975940 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.087453 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:14.088428 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:14.112576 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:14.202805 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:14.232429 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:14.242142 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.248278 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.297212 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:14.363490 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.406186 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.411031 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.415994 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:35:14.501320 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.503082 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.506062 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:35:14.531455 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:14.572470 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:14.577004 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.581060 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:14.582257 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.658079 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.662439 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.690898 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:14.717986 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.759764 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:14.834239 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:35:14.888880 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:14.915603 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.950632 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:14.993039 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.052980 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:15.057236 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.156232 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:15.158866 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:15.180765 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:15.222546 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:15.228128 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.232897 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:15.258045 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 21:35:15.281569 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:15.384965 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:15.532246 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.547350 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:15.560808 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.632766 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:15.634575 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:15.687297 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:15.781445 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:15.792796 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:15.843876 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.844991 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:15.875494 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:15.912552 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:15.974289 22731 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:35:15.974529 22731 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:35:15.975366 22731 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:35:15.978652 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:15.979225 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:35:15.979391 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:35:15.979681 22731 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:15.982549 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:15.992588 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.035907 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.068742 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:16.202286 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:35:16.231094 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:16.292570 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.320075 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.338263 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:16.397796 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:16.412304 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.521734 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:16.543337 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:16.569517 22731 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:16.569734 22731 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:16.570508 22731 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:16.572017 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:16.572237 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:16.572369 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:16.572589 22731 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:16.575399 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.633414 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.670537 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.672984 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:16.782019 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.819070 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.826184 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:16.829547 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:16.888250 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:16.907450 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.912678 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:16.927767 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:16.930651 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.015412 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:17.044767 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.198930 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.229349 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.268710 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:17.295405 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:17.311154 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.315294 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:17.380367 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.390861 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.393419 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.397348 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:17.400727 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.519570 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.528324 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.538888 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:17.561252 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:17.639669 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:17.766116 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:17.808780 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.826332 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.847640 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 21:35:17.851203 23105 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:17.851420 23105 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:17.852087 23105 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:17.853545 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:17.853705 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:17.853874 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:17.854133 23105 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:17.884800 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 21:35:17.885049 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:17.902853 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.916559 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:17.930822 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.099972 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:18.206805 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:18.221194 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:18.255167 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:18.261483 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.288219 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:18.326184 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:18.336797 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.374069 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.431154 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:18.565567 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:18.649991 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:18.700424 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:18.708566 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:18.776525 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.799583 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:18.887943 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.933228 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:18.939824 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.022553 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.100540 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:19.131440 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.177757 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.207640 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:19.210913 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:35:19.235328 22731 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:35:19.235582 22731 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:35:19.236395 22731 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:35:19.237974 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:19.238148 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:35:19.238339 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:35:19.238687 22731 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:19.265334 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.269420 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:19.281298 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.284123 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:19.343927 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:19.355356 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:19.399091 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:19.418411 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 21:35:19.474679 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:19.500751 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.552435 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.604156 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:35:19.673509 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:19.675634 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:19.716980 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:19.801007 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:19.817966 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:19.885159 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:35:19.900871 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:19.909540 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:35:19.914227 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:19.929308 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:19.977056 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:19.977053 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:20.031245 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:20.037355 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.071460 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.139246 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:20.255409 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:20.263854 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:20.284667 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:20.291625 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:20.291764 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:20.317277 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:20.320492 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.448680 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:20.462833 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.555480 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 21:35:20.592278 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:20.654054 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:20.673017 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.673743 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:20.717293 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.733453 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:20.748354 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.757720 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:20.842643 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:20.890086 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:20.898761 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:21.067443 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:21.092507 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:21.187605 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:21.192337 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 21:35:21.232790 23319 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:21.232985 23319 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:21.233861 23319 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:21.235312 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:21.236196 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:21.236328 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:21.236588 23319 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:21.241145 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.248481 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:21.277084 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:35:21.325052 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:21.364612 23319 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:21.364847 23319 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:21.365619 23319 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:21.367331 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:21.367542 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:21.367676 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:21.367957 23319 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:21.413439 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.432680 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.509189 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.535634 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:21.535784 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:21.543685 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.587329 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.608171 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:21.641203 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.643420 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:21.647156 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:21.722234 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:21.822741 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:21.847221 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:21.869740 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:21.917691 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.028245 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:22.040629 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:22.086143 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:22.110700 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.223814 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.236433 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.265753 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.279402 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:22.289460 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.403358 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.458774 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:22.493815 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:22.536432 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.537642 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:22.543407 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:22.608774 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.611868 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.648907 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.795061 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.804898 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.850703 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:22.869654 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:22.902830 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:22.925675 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:22.953543 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.036010 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:23.096930 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.108634 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 21:35:23.112464 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.137341 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:23.161795 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:23.185626 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.305199 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:23.364997 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:23.389148 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:23.441948 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.468690 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:23.632598 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:23.679844 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.711783 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:23.714316 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:23.747396 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:23.849400 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:23.865988 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 21:35:23.881393 23448 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:23.881577 23448 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:23.882300 23448 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:23.883939 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:23.884359 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:23.884488 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:23.884761 23480 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:23.929177 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:23.952375 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.014659 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.030423 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.066769 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:24.067080 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.068228 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:24.083773 23319 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:24.084013 23319 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:24.084779 23319 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:24.086591 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:24.086764 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:24.086885 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:24.087143 23319 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:24.090234 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:24.100733 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.199626 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:24.206429 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.318158 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:24.321105 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:24.367354 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.432451 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:24.477283 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:35:24.504462 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:24.593024 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:24.636471 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:35:24.640192 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:24.657217 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:24.681706 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.705869 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.775513 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.791003 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:24.853251 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:24.856441 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:24.898696 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:24.899533 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:24.926625 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 21:35:24.996217 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.023186 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:25.038514 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.040266 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:25.143771 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.164363 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:25.239814 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.268888 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:25.322010 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:25.369482 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.438566 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:25.440639 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:25.442760 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:25.475186 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.477769 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.486699 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.559998 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:25.604161 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:25.653851 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 21:35:25.742821 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.801297 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:25.828435 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:25.845705 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:25.939886 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:25.949043 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:26.029548 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.031993 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:26.105973 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:26.118314 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.153303 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:26.263487 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:26.266965 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.369684 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:35:26.370632 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.408478 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:26.413334 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.441805 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:26.472936 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.506124 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:26.510553 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.517546 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:26.535622 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:26.584652 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.646101 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:26.757477 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:26.804796 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.861392 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:26.909351 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:26.946466 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.045970 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:27.055562 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.089622 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:27.090762 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.092061 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:27.115636 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.147300 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.156633 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:27.243593 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:27.354864 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.528338 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:27.543063 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.551226 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.615278 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:27.616025 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:27.627609 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:27.629005 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.725450 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.733008 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.752223 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.777915 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:27.778369 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.936339 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:27.949833 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:27.959791 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.006688 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.069821 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.081817 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.131460 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.143092 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:28.209784 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:28.240130 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.305163 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:35:28.310266 23319 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:35:28.310503 23319 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:35:28.311086 23319 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:35:28.312763 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:28.313130 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:35:28.313254 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:35:28.313645 23319 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:35:28.335544 23319 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:28.335745 23319 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:28.336380 23319 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:28.336602 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 21:35:28.337174 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:28.337515 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:28.337648 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:28.337899 23319 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:28.390225 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.420202 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.504154 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.513938 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:28.544042 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:28.544238 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.623757 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.640233 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.745687 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.756754 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:28.791227 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.794605 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:28.799199 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.916043 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:28.970141 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:28.997301 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:29.051959 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:29.072991 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:29.076622 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:29.170635 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:29.201484 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:29.217007 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:29.254798 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:29.287992 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:29.367918 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:29.372156 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:29.397703 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:35:29.423710 23742 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:29.423923 23742 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:29.424768 23742 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:29.426417 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:29.426710 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:29.426797 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:29.427031 23742 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:29.427851 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:29.527316 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:29.540706 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:35:29.589879 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:29.620324 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:29.633119 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:29.658838 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:29.663498 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:35:29.705238 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 21:35:29.780166 23741 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:29.780447 23741 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:29.781178 23741 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:29.783208 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:29.783735 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:29.783874 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:29.784133 23741 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:29.806929 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:29.851824 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:29.982265 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:35:30.000803 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:30.031270 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.113749 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:30.128477 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:30.149531 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.159312 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.209543 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:30.247716 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.281469 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:30.297782 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.305163 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.371454 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.408586 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.414142 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.473340 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.484015 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:30.587867 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.590543 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.594278 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.637230 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.685595 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:30.730917 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:30.755697 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.810837 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 21:35:30.881165 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:30.907080 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.937567 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:30.979285 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:31.000844 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.024737 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:31.095899 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.104651 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:31.193372 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:31.220850 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:31.325294 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.348004 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:31.416915 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:31.452814 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.480778 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.501567 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:31.520094 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:35:31.521176 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:31.532480 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:31.555769 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:31.560019 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.636561 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:31.707924 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:31.786325 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.831266 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:31.911403 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:31.957072 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:31.963199 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.083631 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:32.085768 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.118312 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:32.144310 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:32.144310 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:32.150732 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:32.373723 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:32.421576 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.483074 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.524385 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:32.554819 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.564417 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:32.567773 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:32.570583 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:32.622905 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:32.642164 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:32.725975 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.730808 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:32.799229 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.852488 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:32.883317 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:33.001890 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.006964 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.011025 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.017391 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:33.041239 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.045146 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.046317 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:33.123414 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:33.147629 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.153419 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:33.153653 23741 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:33.153824 23741 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:33.154628 23741 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:33.156368 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:33.156592 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:33.156746 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:33.157023 23741 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:33.363415 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:33.446851 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.472416 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.531323 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:33.541857 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 21:35:33.595857 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:33.620754 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.628877 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:33.642084 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:33.671907 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.736392 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:33.747486 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:33.766197 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:33.865026 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:33.870465 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:34.035545 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.069654 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:34.117995 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.136550 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.136576 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:34.142468 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.154457 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:34.239218 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:34.276904 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:34.288003 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:34.343890 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:34.535815 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.550720 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:35:34.574512 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:34.590135 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:34.609416 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.647694 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:34.696180 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.721530 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:34.746873 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:35:34.751890 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:34.862576 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:34.878039 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:34.888530 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.918182 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:34.978086 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:35.025817 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240502 21:35:35.036267 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:35.055425 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:35.091271 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:35.095065 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.156098 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:35.196807 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:35.203495 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.273469 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:35.298548 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:35.363611 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.372761 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.411203 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:35.543041 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.554905 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.593129 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:35.602205 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:35.617269 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:35:35.660679 24024 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:35.660938 24024 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:35.661861 24024 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:35.662379 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:35.663298 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:35.663548 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:35.663667 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:35.663982 24024 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:35.672241 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 21:35:35.687520 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:35.703007 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.771863 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.789232 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:35.833894 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.861627 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:35.890974 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:35.983680 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:36.067991 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:36.067991 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.094408 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:36.141654 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:36.149221 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:36.164059 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.235710 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.263124 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:36.376065 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:36.458117 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:36.492127 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:36.516969 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:36.546250 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 21:35:36.605098 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.612725 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.617411 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:36.654505 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:36.698632 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.793702 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:36.836274 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:36.839746 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:36.944926 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:37.022528 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.063969 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.074885 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:37.100041 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.147012 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 21:35:37.167376 23741 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:35:37.167586 23741 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:35:37.168354 23741 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:35:37.170006 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:37.170229 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:35:37.170401 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:35:37.170693 23741 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:37.195367 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.197317 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.219419 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:37.230633 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.231415 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.309691 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:37.337472 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.349556 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.435894 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:37.492964 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:37.516680 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.575230 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:37.635593 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.636070 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:37.646234 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.701414 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:37.813360 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:37.912508 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:37.945518 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:37.994099 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:38.005018 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.021875 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:38.033623 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:38.054203 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.118682 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.122077 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:38.144026 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.166846 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.280038 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.335381 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:38.349447 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 21:35:38.393232 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.396160 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:38.482149 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.496223 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:38.515956 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:38.546664 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:38.593791 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:38.627059 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.700486 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:38.741461 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:38.815807 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:38.829974 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:38.939770 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:38.995471 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.017526 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:39.097404 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.125799 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.170583 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:39.174022 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.265038 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.307122 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.383366 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.415756 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.489675 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.498591 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.504216 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:39.519438 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.572996 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:39.591863 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:35:39.602829 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:39.611294 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.628365 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:39.744395 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.766572 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.837810 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:35:39.867101 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:39.891604 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.920766 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:39.964393 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:39.976982 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:40.020081 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.024323 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:40.052490 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240502 21:35:40.053442 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:40.096606 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:40.133863 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.145759 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:40.279772 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.284343 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.305047 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.348811 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:40.389456 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:40.390157 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:40.437785 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:40.511101 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:40.544792 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:40.567725 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.659600 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.664662 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.665021 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.669226 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:40.686008 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.699790 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 21:35:40.802903 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:40.853957 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:40.862257 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:40.868468 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:40.875078 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.002271 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:41.051172 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.096004 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.117822 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:41.182049 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.239490 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.264164 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.276343 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:41.281839 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:41.339612 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.465343 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.489714 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:41.489742 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:41.547601 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:35:41.599622 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.681061 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.708182 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.737062 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:41.769793 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:41.790735 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.822450 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.852325 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:41.898742 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:41.999008 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.003208 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.003572 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.017302 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.074903 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.211318 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:42.225731 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.251278 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:42.274178 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:42.318279 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:42.327420 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.345633 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.448331 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.499019 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:42.507530 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.594496 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.605966 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.627333 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.672943 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:42.733515 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:42.747043 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.781322 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:42.802650 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:42.822875 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 21:35:42.866465 23741 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:42.866703 23741 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:42.867646 23741 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:42.869092 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:42.869379 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:42.869540 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:42.869666 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:42.869932 23741 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:42.911249 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.950076 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:42.958527 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:43.062309 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.082191 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.134379 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:43.216014 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 21:35:43.218001 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:43.253958 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.302582 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:43.358227 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:43.382674 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.455597 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.457085 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.510334 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:43.542555 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:43.612051 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:43.629504 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:43.716197 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:43.729614 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:43.776433 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:43.792575 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:43.806005 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.843214 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:43.919808 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:43.920393 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.098443 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:44.128501 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.129375 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:44.153956 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:44.229859 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:44.257894 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:44.260855 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.327214 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.416451 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:44.482028 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.485015 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.535727 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:44.598335 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:35:44.621740 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:44.623456 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.682803 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:44.752959 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:44.763826 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.794637 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:44.815169 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:44.832856 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:44.902827 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:35:44.903803 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:44.906147 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:44.919116 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:45.054229 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:35:45.094327 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:45.160926 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:45.161703 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.221427 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.266531 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:45.267279 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.284055 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:45.292964 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:45.307688 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:45.348002 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:45.484506 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 21:35:45.503252 23741 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:45.503448 23741 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:45.504107 23741 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:45.505934 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:45.506079 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.506162 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:45.506278 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:45.506565 23741 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:45.522199 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:45.536254 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.578120 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.602109 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:45.764124 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 21:35:45.782864 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.783190 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:45.835691 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:45.877882 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:45.897204 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:45.912452 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:46.025945 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.045289 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:46.076828 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:46.094910 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:46.125497 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.164693 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.192999 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:46.254418 23741 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:35:46.254603 23741 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:35:46.255352 23741 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:35:46.256572 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:46.256780 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:35:46.256877 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:35:46.257086 23741 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:46.274304 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:46.300415 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:46.327814 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:46.354843 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:46.370116 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.496059 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:46.525100 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:46.532918 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.560959 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:35:46.571065 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:46.711262 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.759004 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:46.807122 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:46.835592 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.847826 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:46.871801 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:46.897320 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.018913 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.019053 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.026060 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:47.064962 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.091466 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:47.101804 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.136514 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 21:35:47.137735 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:47.323257 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:47.329676 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.350968 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.378691 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:47.386507 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.402195 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:47.435824 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.519248 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.542099 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:47.597873 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.642933 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:47.716377 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.720950 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:47.814311 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.818596 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.820170 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:47.825188 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.828274 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:47.860270 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:47.948905 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:47.968222 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.007159 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.010229 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.040343 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:48.154116 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:48.196159 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:48.232529 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:48.272150 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 21:35:48.351261 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:48.402184 25180 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:48.402402 25180 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:48.403108 25180 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:48.404356 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:48.404518 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:48.404639 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:48.404860 25180 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:48.417145 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.442111 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:48.474860 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.504321 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:48.520570 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:48.629216 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.638307 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:48.640334 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:48.738077 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:48.748180 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:48.753430 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.757719 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.809187 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:48.913172 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:48.922387 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:49.008131 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:49.086791 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:49.143651 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.217784 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:49.329253 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.336597 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:49.361035 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.363802 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:49.407701 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.414183 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:49.420439 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:35:49.478688 23741 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:49.478955 23741 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:49.479691 23741 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:49.481240 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:49.481431 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:49.481559 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:49.481827 23741 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:49.517858 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:49.577219 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:49.605294 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:49.605983 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.615823 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 21:35:49.616869 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:49.651269 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.761622 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:49.769104 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:49.770778 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:49.791412 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:49.857300 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:49.868408 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:49.937042 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:49.947165 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:35:49.964838 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:49.969988 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:50.120489 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:50.121934 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 21:35:50.171303 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.209705 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:50.218302 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:50.284991 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:50.310427 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.332798 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:50.339764 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 21:35:50.407967 25340 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:50.408218 25340 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:50.408949 25340 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:50.409459 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.410264 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:50.410491 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:50.410583 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:50.410809 25340 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:50.434718 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:50.454156 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.537369 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:50.550650 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.561743 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:50.700423 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 21:35:50.738873 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.740656 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:50.759864 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.797950 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:50.860502 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:50.872042 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:50.905120 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.913699 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:50.939296 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:50.973481 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:50.984453 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.017051 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:51.204749 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.213271 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.241155 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:51.265255 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.274623 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.284214 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:51.377863 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.412066 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.443056 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.446516 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.476768 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:51.588529 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:35:51.589478 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:51.703014 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.813364 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.840361 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.881742 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:51.914762 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.935375 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.935492 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.948817 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.956560 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:51.959172 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:51.976891 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.014386 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.056766 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:52.082831 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:52.101811 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:52.182235 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:52.203670 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.239964 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:52.311820 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:52.364604 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:52.483922 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.595314 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:52.616690 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:52.624691 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:52.629957 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:52.769304 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.775197 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.806056 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.843099 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:52.843708 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:52.892336 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:52.943944 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:52.998598 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:53.005681 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:53.171520 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:53.239856 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:53.266839 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.300967 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.310916 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.373412 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 21:35:53.374743 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:53.379662 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.466063 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:53.494115 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.505977 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:53.516180 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:53.553030 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:53.563282 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 21:35:53.591542 25392 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:53.591784 25392 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:53.592741 25392 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:53.594880 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:53.595083 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:53.595189 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:53.595486 25392 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:53.668746 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:53.702028 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.731808 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:53.749596 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:53.772159 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:53.936621 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.037377 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.050065 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:54.057391 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.066213 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.173928 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:54.211323 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:54.259156 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.262840 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.263738 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.274765 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.319618 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.332973 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.347821 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.470166 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:54.477344 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.492270 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:54.511584 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:54.528470 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.528580 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.556684 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:54.625188 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:54.636435 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 21:35:54.651472 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:54.678176 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:54.879498 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.919498 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.973703 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:54.996783 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:35:55.097016 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:55.111465 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:55.122314 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:35:55.221117 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.221477 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.248160 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:55.288782 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.403437 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.422082 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.436183 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.456552 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:55.478839 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:55.486368 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:55.520011 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.529153 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:55.557178 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:55.573230 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.670743 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:55.701572 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:55.826277 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:55.832005 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 21:35:55.885653 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.902635 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.912568 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.957597 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:55.959092 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:55.992486 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:56.018568 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:56.047269 25392 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:56.047516 25392 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:35:56.048234 25392 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:56.050384 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:35:56.050621 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:56.050719 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:56.051080 25392 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:56.132668 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.134583 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:56.160308 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:56.252887 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:56.274820 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:56.420639 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.539327 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:56.543597 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:56.549896 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.607806 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:56.608453 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:56.644889 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:56.654647 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.662889 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:35:56.682690 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:56.696354 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.876288 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.897814 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.924986 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:56.959635 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:56.974500 25392 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:56.974694 25392 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:35:56.975471 25392 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:56.977249 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:56.977530 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:35:56.977659 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:35:56.977963 25392 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:56.996806 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:57.000859 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.054689 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.065418 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:57.082154 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.085438 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:57.096563 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.155592 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.236541 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.249820 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.272258 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:57.366266 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:57.509756 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.547389 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:35:57.626610 25629 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:35:57.626783 25629 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:35:57.627490 25629 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:35:57.628764 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.628890 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:35:57.628988 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:35:57.629173 25629 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:35:57.707602 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.713425 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:57.734936 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:57.774158 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:57.783805 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.832093 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.865185 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.866072 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:57.866914 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.907801 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:57.956487 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:57.957106 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:57.983129 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.011703 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.067420 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:58.068280 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 21:35:58.124617 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:35:58.170687 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.306255 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:58.343561 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:58.346503 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:58.375185 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.389639 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:58.396958 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:58.479416 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.495991 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.508545 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.540433 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:58.551963 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:58.784961 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:58.812094 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:58.845867 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:58.860749 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:58.890947 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:58.915237 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:59.033203 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.222060 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:35:59.222877 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.224604 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.285732 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:59.307950 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.354534 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.380329 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:59.388401 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:59.394474 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.403406 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:59.417153 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:35:59.421689 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.428846 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.461222 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.466146 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:59.481160 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:35:59.559103 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:35:59.597541 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:35:59.647954 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:35:59.695420 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:35:59.795763 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:35:59.819605 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:35:59.821470 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:35:59.883594 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.013399 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.055388 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:36:00.082902 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.094463 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.132269 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 21:36:00.133155 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:00.237095 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.244038 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.248179 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:00.258289 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:00.366267 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:00.402240 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:00.415995 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.421970 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.494732 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.499910 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.617389 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 21:36:00.642256 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.695745 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.720572 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.740384 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:00.745787 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.783370 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.801231 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:00.813879 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:00.862221 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.933529 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:00.990798 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:00.999294 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:01.000589 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:01.022907 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.081455 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.247223 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:01.328145 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:01.355528 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:01.383148 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.402334 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:01.585853 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:01.592964 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.613860 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:01.674618 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.675441 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:01.814839 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.830893 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:36:01.866844 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:01.904947 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.926826 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:01.942085 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.949221 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.950549 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:01.961680 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:01.969563 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:02.014504 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.071799 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:02.084739 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.102913 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.105743 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.155066 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:02.216449 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:02.268021 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:02.473119 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:02.487771 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:02.507858 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:02.579900 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:02.590875 25392 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:02.591104 25392 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:02.591950 25392 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:02.593652 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:02.593813 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:02.593930 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:02.594219 25392 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:02.631245 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.722061 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.734867 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:02.809602 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.840723 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:02.872658 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:02.899891 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:02.928539 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:02.958235 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:02.976974 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.042745 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:03.043265 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:03.086784 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.091537 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.118969 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:03.159103 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.192863 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 21:36:03.348763 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:03.374225 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:03.385946 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:03.433303 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.479952 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:03.484169 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.500370 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.534794 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:03.543524 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:03.572168 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.572304 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:03.601576 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:03.870992 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:03.916549 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:03.948066 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:03.952111 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:04.003158 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:04.014086 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:04.144477 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.160357 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.252218 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 21:36:04.286881 25392 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:36:04.287120 25392 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:36:04.288391 25392 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:36:04.290329 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:04.290652 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:36:04.290769 24718 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:36:04.291216 25392 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:04.296103 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.333678 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:04.349820 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.401955 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.403317 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:04.410612 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:04.412982 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:04.435966 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.449942 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.470372 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:04.472759 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.509284 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:04.513010 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:04.525874 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:04.565034 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:04.609099 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.705765 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:04.856902 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:04.866394 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 21:36:04.918889 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:04.921300 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:04.926326 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.014307 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.036063 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:05.099357 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:05.142843 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:36:05.146063 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:05.169021 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:36:05.171311 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.298799 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:05.335382 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:05.356519 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.485407 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:05.506742 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:05.583994 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:05.589618 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:05.600684 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.616868 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.639755 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 21:36:05.643931 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:05.703341 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.729043 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:05.818898 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.894724 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:05.901337 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:05.901738 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:05.905725 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:05.910775 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:06.056768 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.128334 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.167968 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.204624 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.267547 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.280171 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:06.306196 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.388257 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.419855 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:06.492714 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.510679 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:06.511723 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.533468 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.655539 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.666647 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.672149 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:06.699682 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 21:36:06.718000 26097 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:06.718195 26097 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:36:06.719035 26097 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:06.720754 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.721192 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:06.721300 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:06.721575 26097 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:06.728610 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.730746 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:06.812702 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.830261 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.843811 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:36:06.896303 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:06.914455 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:06.917038 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.945676 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:06.964018 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:07.012478 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:07.036640 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.093173 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.096238 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:07.122113 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:07.233299 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:07.285857 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.354475 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.467469 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.524257 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:07.525626 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:07.535118 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.574692 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:07.624135 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:07.645848 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:07.708489 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:07.808102 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:07.831629 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:07.973724 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.023643 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:08.045387 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:08.103215 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 21:36:08.105275 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.127699 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:08.133251 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:08.148686 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:08.235029 25392 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:08.235237 25392 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:08.235889 25392 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:08.237871 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:08.238250 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:08.238370 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:08.238646 25392 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:08.245167 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.261240 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:08.338853 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:08.351598 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:08.400120 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.418192 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.459208 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:08.528687 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.530860 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:08.531984 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:08.607364 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:08.710773 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:08.763921 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.776481 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:08.799743 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:08.885744 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:08.971459 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:09.001397 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.020001 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:09.048514 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.071089 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.083851 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:09.090327 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:09.115657 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.187604 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:09.272603 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.297307 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.365993 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.398516 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:09.403873 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:09.406211 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.442414 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.455853 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:09.505019 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.575469 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:09.616295 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:09.624410 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:09.630245 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:09.649555 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:09.686193 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.712832 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:09.824843 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:09.831899 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:09.847319 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:09.848695 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:09.855607 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:09.911633 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:36:09.943961 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:10.135198 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:10.147218 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:36:10.163302 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:10.176271 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:36:10.201324 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:10.247864 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:10.315482 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:10.387136 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:10.400703 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:10.502514 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 21:36:10.542727 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:10.667496 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:10.695436 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:10.713723 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:10.713917 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:10.781109 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:10.792462 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:10.825095 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:10.851228 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:10.868757 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:10.917757 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:10.924855 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:10.958878 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.085301 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.138532 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:11.192767 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.204833 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:11.213330 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:11.273499 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:11.399384 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.426384 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.438616 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:11.512041 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:11.561467 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.603436 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:11.604799 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:11.615216 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.645151 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:11.683559 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.697199 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:11.707063 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.740878 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.860145 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.872655 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.904983 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:11.905022 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:36:11.905951 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:11.947965 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:11.978315 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.034686 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:12.041319 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:12.044728 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:12.056452 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:12.137845 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:12.158524 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.204483 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.223986 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:12.264297 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.310151 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.322531 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:12.374415 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.496304 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:12.525600 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:12.527698 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.568759 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:12.579144 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:12.768003 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.930161 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:12.949692 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:12.960669 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:12.992272 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:13.012495 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:36:13.018592 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.018689 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.023891 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.100554 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.214598 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:13.299158 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:13.392590 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:13.405380 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:13.448356 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:13.461745 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.498152 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.508558 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:13.517421 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.550822 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.562088 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.648380 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.811870 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.844682 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:13.860055 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:13.913697 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.938457 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:13.938932 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.001780 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.019508 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.028131 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.062114 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:14.106187 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:14.121362 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:14.131525 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.198680 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.313287 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.365028 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.399212 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.401710 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:14.423797 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.426160 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.479266 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.523855 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.552513 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:14.573817 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.643014 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:14.667963 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:14.774201 25392 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:14.774417 25392 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:36:14.775156 25392 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:14.776497 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:14.776665 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:14.776779 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:36:14.777104 25392 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:14.783891 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.798437 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:14.844311 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:14.847299 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:14.890328 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:14.894948 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.910765 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.925493 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:14.934288 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:36:14.936234 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:14.988453 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:14.988744 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:15.072088 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:15.086423 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:15.179709 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:36:15.215781 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:36:15.234010 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:15.363426 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:15.423179 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:15.446074 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:15.507426 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:15.511360 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:15.602205 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:15.646670 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:36:15.702435 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:15.748786 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:15.759557 25392 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:15.759761 25392 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:15.760421 25392 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:15.762261 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:15.762466 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:15.762543 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:15.762799 25392 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:15.776187 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:15.806047 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:15.810782 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:15.873824 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:15.944604 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:15.949122 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:15.966336 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:16.114212 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:16.118153 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.125475 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.132073 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.145715 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:16.151079 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.332067 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.365343 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:16.377207 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:16.384666 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.395382 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:16.431847 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:16.534016 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:16.550590 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.633773 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:16.636796 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.658881 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.676295 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.720973 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.727320 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:16.805456 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:16.834718 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.840212 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.861196 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:16.933343 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:16.961218 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:16.981303 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:16.981990 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 21:36:17.041481 25392 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:17.041733 25392 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:36:17.042511 25392 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:17.044066 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:17.044273 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:17.044407 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:36:17.044693 25392 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:17.091672 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:36:17.103399 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:17.160315 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:17.172048 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:17.196660 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.215865 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:17.292569 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:17.341193 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:17.400817 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:17.486326 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.492595 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.496140 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:17.526316 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.564996 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.575726 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.595728 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:17.673782 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:17.696048 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:17.948869 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:17.963059 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:17.974622 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.046931 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:18.118541 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:18.136401 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:18.175029 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:36:18.211932 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:18.268865 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.392832 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:18.448879 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:18.501104 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:18.537518 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:18.588654 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:18.605232 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:18.631896 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.653194 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:18.660760 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:18.702391 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.729120 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.730859 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.844188 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.936080 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:18.954833 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:18.986399 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.062117 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:19.099987 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.106940 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:19.112783 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.119357 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:19.119606 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.153458 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.176169 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:19.203891 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.239362 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:19.244237 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.266325 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.297950 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.333523 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.368961 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.376152 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.533843 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.593334 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:19.597939 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:19.607671 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:19.654181 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.682325 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:19.736500 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.744931 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.920100 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:19.933876 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:36:19.946552 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:36:19.948175 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:19.979038 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.029065 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:20.047338 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.069051 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.069236 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.084915 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 21:36:20.092339 25392 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:36:20.092569 25392 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:36:20.093308 25392 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:36:20.094870 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:20.095023 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:20.095153 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:36:20.095423 25392 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:20.196609 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240502 21:36:20.210943 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:20.224756 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:36:20.244392 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:20.329583 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:36:20.340679 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.342602 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:20.354781 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.405203 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:20.547950 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:20.607895 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:20.696008 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:20.702068 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:20.765129 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:20.782265 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.900730 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:20.912469 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:20.930713 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:20.964870 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:20.999025 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.007699 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:21.092083 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.147464 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:21.169214 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.177175 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:21.232810 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:21.294128 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.381707 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:21.384209 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:21.431565 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.474947 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:21.482278 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:21.519181 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:21.541474 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.556711 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.586599 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:21.605773 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.609040 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.616186 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:21.679912 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.805248 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:21.832922 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.843972 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.853552 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:21.861208 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:21.890545 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:21.921198 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:21.936055 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:22.032305 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:22.077406 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:22.101944 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.115465 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:36:22.179734 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.187687 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:22.227051 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.387171 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:22.400432 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:22.467301 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.472729 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:22.477558 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:22.489044 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.628247 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:22.629848 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.631127 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:36:22.656980 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.669997 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:22.804534 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.900743 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:22.918879 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:22.922412 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:22.994127 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:23.144611 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:23.155207 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:23.195679 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:23.203291 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.228811 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:23.334625 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.425297 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:23.431815 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:36:23.436863 26911 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:23.437049 26911 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:36:23.437805 26911 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:23.439406 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:23.439738 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:23.439854 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:23.440101 26911 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:23.452291 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:23.502740 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.539301 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:23.544381 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:23.581202 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:23.589725 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:23.688938 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:23.865679 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.896807 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.902597 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:23.928973 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.964603 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:23.977488 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:23.983783 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:23.995596 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.011427 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:24.037703 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.080873 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:24.114003 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:24.123584 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:24.157454 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.180307 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:24.196018 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.341944 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.352224 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.354413 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.362226 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:24.521956 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:24.534658 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.557495 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:24.598045 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:24.661403 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:24.673079 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:24.699271 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:24.719403 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:24.736909 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:24.827476 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:24.916411 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:24.948225 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:24.973182 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:25.119177 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:36:25.122009 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:25.152746 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:36:25.181811 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:25.198197 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.220717 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 21:36:25.221642 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:25.264489 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:36:25.280333 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:25.297916 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.345144 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.350070 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:25.376425 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.607654 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.680985 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:25.691792 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:25.693675 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:25.696177 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:25.700738 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:25.704774 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:25.815575 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:25.836980 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.895639 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:25.998637 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:26.108433 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.141402 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:26.150893 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:26.156800 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:26.201776 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.209475 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:26.212082 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:26.237576 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:26.254045 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:26.262900 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 21:36:26.269495 26911 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:26.269692 26911 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:36:26.270390 26911 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:26.271885 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:26.272050 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:26.272138 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:26.272356 26911 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:26.280591 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.478668 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.481415 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.501902 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.530732 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:26.564666 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:26.593369 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.616034 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.646975 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:26.672060 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:26.682269 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.736706 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.761700 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.782523 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:26.846159 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:26.874847 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:26.908672 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:26.998541 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:27.058161 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:27.069166 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:27.080806 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.092504 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:27.105749 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:27.147817 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:36:27.187170 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.197408 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:27.386217 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:27.440339 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:27.518260 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:27.574080 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:27.589852 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:27.598291 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.610916 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:36:27.638998 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.658869 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:27.825171 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.831084 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.856901 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:27.857046 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:27.908476 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:28.012499 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:28.043087 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:28.055445 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:28.141175 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:28.142431 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:28.198087 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:28.248097 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:28.354732 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:28.406235 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:28.432001 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:28.443557 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:28.459749 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:28.483991 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:28.597903 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:28.604777 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:28.640054 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:28.658720 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:28.661130 27017 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:28.661327 27017 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:28.662107 27017 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:28.663936 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:28.664104 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:28.664193 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:28.664429 27017 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:28.719187 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:28.729133 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:28.818320 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:28.823549 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:28.936844 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:28.978945 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.055418 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.088703 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.088985 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:29.092278 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.165243 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:29.171056 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.190657 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.247555 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:29.262459 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.276710 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.281850 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:29.372440 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:29.396073 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.415165 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:29.459350 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.481163 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.493546 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:29.554477 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:29.567824 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:29.589674 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:29.678833 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:29.798619 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:29.816336 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:29.871969 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:29.944600 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:36:29.951380 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:29.952265 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:30.113595 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:30.149747 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:36:30.160815 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.194871 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:30.234108 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:36:30.236325 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.241122 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:30.244509 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.391405 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.463089 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 21:36:30.464664 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.467308 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:30.483917 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:30.491958 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:30.511258 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:30.528635 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.591001 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:30.616252 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:30.754805 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:30.801283 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:30.862085 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:30.929468 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:30.965430 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:30.982481 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:31.010206 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:31.025076 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.039604 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.082173 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:31.102234 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:36:31.151368 27260 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:31.151587 27260 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:36:31.152351 27260 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:31.153882 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:31.154073 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:31.154188 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:31.154476 27260 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:31.155392 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.234206 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.300738 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:31.355412 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:31.359261 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:31.393406 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 21:36:31.404585 27017 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:31.404803 27017 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:31.405571 27017 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:31.407291 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:31.407488 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:31.407559 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:31.407763 27017 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:31.430416 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:31.466323 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:31.493899 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.519368 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.548229 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.628289 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.636309 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:31.677556 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.689478 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.699381 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:31.788429 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.812048 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.845093 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:31.892365 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:31.913769 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:31.918018 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:31.962002 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:32.073024 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.076102 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:32.156265 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:36:32.157063 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:32.172327 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.179077 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.221573 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:32.257450 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:32.356472 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:32.377408 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:32.393600 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.404325 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:32.472095 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:32.478570 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.636391 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:32.664886 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.695523 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:32.756054 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:36:32.761996 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.788650 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.865278 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:32.922518 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:32.923429 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:32.993903 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.016010 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.030820 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:33.073201 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.094434 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:33.187038 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:33.374012 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.436985 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:33.453191 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.530970 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.545943 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:33.571319 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:33.588208 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.689711 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:33.703671 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:33.708773 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.724184 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:33.729010 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.781898 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.824090 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.935206 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:33.935312 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:33.942667 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.973433 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:33.996649 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:34.004972 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:34.052052 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.122871 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.140528 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.167057 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.266741 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:34.314719 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.354382 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.359807 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.385462 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:34.425705 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:34.437574 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:34.489631 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:34.496835 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:34.525372 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.527455 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:34.568071 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.596180 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:34.698550 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:34.718974 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:34.785063 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:34.912361 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:34.951136 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:34.996632 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.000371 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.048692 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.073236 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:35.108608 27017 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:35.108824 27017 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:36:35.109616 27017 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:35.111049 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:35.111689 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:35.111814 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:36:35.112005 27017 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:35.132977 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.169858 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:35.207151 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:36:35.208029 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:36:35.242942 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 21:36:35.250015 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.290169 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.317078 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:35.400264 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:35.449362 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.458261 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.462461 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.467357 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.484436 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:36:35.542930 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.568879 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:35.773859 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:35.812932 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:35.836053 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.963712 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:35.978884 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:35.979502 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:36.035496 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:36.056975 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:36.075023 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.078090 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.110682 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.127220 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:36.161209 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:36.305119 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:36.310539 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:36.369582 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.383634 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:36.390480 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:36.394634 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:36.483736 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:36.562776 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.579587 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.631942 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.683179 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.777086 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.809423 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.819444 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.830212 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:36.838248 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:36.870337 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:36.885429 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:36.892613 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:36.960947 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:36.985666 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:37.044958 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:37.075557 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:37.081543 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.095883 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:37.227742 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.251650 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:36:37.260517 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.290856 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:37.389192 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:37.399158 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:37.408689 27017 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:36:37.408896 27017 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:36:37.409576 27017 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:36:37.411003 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:37.411159 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:37.411266 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:36:37.411514 27017 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:37.435181 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.500406 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:37.520877 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:36:37.576570 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:37.582118 27481 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:37.582293 27481 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
W20240502 21:36:37.582609 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 21:36:37.582988 27481 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:37.584385 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:37.584569 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:37.584677 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:37.584931 27481 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:37.636533 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.705053 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.763041 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:37.787902 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:37.809393 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:37.816617 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:37.890264 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:37.918256 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.057171 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.070977 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.129491 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:38.153981 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.249732 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:38.300128 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:38.335203 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.335491 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:38.374250 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.431461 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.464843 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.507071 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.520140 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.582492 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:38.677392 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.698683 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:38.762620 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:38.789220 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:38.808876 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:38.850232 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:38.891474 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.934461 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:38.946938 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:38.958719 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.015530 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:39.093385 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.098155 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.182147 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.194684 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:39.258456 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:39.268756 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.338649 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:39.379945 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:39.413060 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.417570 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.428336 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.448573 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.457304 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:39.502983 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:39.589308 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:39.603302 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:39.650607 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:39.664230 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.809345 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:39.866072 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:39.869477 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:39.919636 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:36:39.949496 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:39.989231 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:40.004235 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:40.006058 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.165745 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:40.257651 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:36:40.260874 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.330466 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:40.342675 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 21:36:40.358944 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.374873 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:40.410820 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:40.412900 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:40.433136 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.465718 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.487382 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:36:40.528450 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:40.531266 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.589179 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:40.631704 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.732417 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:40.818048 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:40.896131 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.953853 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:40.967041 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:40.990556 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:41.065011 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:41.069818 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:41.073207 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:41.159802 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:41.182637 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:41.185457 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:41.193065 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.203027 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:41.296509 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:41.392136 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.458034 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:41.472185 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.499495 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.540203 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:41.607934 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.702303 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.713387 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:41.728664 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:41.730811 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:41.731607 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.789302 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.800256 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.867548 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:41.905839 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.906806 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:41.929131 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:42.013756 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:42.041559 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:42.047009 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.077967 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.225828 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:42.264302 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:36:42.301493 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:42.303314 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:42.321573 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.408130 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:42.443028 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:36:42.452309 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.459707 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:42.465214 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:42.469723 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.526063 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 21:36:42.553122 27859 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:42.553371 27859 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:42.554276 27859 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:42.561293 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:42.561525 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:42.561609 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:42.561897 27859 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:42.604311 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:42.651758 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:42.713754 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:42.847728 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:42.854315 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.873417 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:42.918913 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:42.927723 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.011240 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.045464 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:43.065413 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:43.067121 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.177640 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:43.231221 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:43.249086 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.292629 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:43.479614 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:43.491457 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.522580 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:43.561578 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:43.575960 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:43.581071 27945 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:43.581259 27945 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:36:43.581995 27945 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:43.583333 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:43.583501 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:43.583626 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:43.583899 27945 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:43.613409 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.627166 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:43.730777 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:43.794811 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.799994 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:43.816787 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:43.822557 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.865207 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:43.971446 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:43.987490 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:43.997700 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:44.004135 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:44.009689 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.148149 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.168368 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:44.189898 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:44.206662 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.268226 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.271009 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:44.281003 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.288092 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:44.295475 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.337774 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.532133 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:44.607112 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:44.611524 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:36:44.613766 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:44.661974 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:44.700875 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:44.755064 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.759881 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:44.849452 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:44.865989 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:44.980742 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:44.990196 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:44.993098 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.057734 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.093842 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.148496 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.153259 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:45.247656 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:45.285882 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:36:45.289211 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.293277 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:45.360985 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.397302 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.419729 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 21:36:45.420614 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:45.529388 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:36:45.529533 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.530246 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.533483 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.559988 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.569600 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.594637 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:45.723568 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:45.734119 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:45.784508 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 21:36:45.826913 27945 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:45.827203 27945 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:36:45.827980 27945 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:45.829372 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:45.829598 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:45.829701 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:45.829950 27945 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:45.917714 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:46.019651 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.048256 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:46.114773 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:46.128713 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:46.179597 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.255753 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:46.275539 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.279593 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:46.337025 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.345927 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:46.353458 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.368834 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:46.395567 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.408689 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:46.459057 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:46.464155 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.475107 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.536453 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.557071 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:46.565003 27859 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:46.565225 27859 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:36:46.565951 27859 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:46.567456 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:46.567849 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:46.567934 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:36:46.568156 27859 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:46.596771 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:46.625617 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:46.686445 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.804512 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:46.835876 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:46.848093 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.881531 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:46.917204 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 21:36:46.965044 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:46.974479 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:46.977811 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:47.040299 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:47.041553 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:47.168623 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 21:36:47.169847 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 477d9df7d3a74b8584f3669be3256e00: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:47.178592 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.233448 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:47.281479 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:47.400581 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:36:47.441043 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:47.457873 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.493050 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:47.510816 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.556259 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:47.562616 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:47.592350 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.595402 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:47.654405 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.670712 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.703644 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:47.802279 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:47.808569 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.833196 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.927865 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.968119 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:47.995661 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:48.008332 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.020051 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.054903 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.073451 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.178162 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:48.236555 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:48.290621 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:48.349474 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.485072 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.491091 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.509071 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.667780 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.677059 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:48.678805 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:48.699518 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:48.715744 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.725159 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:48.739856 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.773172 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:48.810777 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:48.826244 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.856026 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.863291 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.893262 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:48.919644 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:48.940604 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.098202 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.098214 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.122696 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.165462 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:49.176091 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:49.249375 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.251178 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.453763 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.470089 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.472637 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 21:36:49.510411 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.522984 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:49.543164 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:49.609308 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.646656 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.716490 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:49.725351 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:49.785079 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d8a25580bc74004b0e31c34f841d2cc: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:49.833628 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:49.862620 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:49.897074 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:49.901415 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.909368 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:49.935693 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:49.961745 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.025691 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:50.181277 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:50.186030 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:50.221685 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.249241 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:50.251339 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.327327 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:36:50.344336 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.351521 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.421684 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240502 21:36:50.422607 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:50.425316 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:50.453399 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.523020 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:50.543462 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:50.573166 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:50.605515 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:36:50.606562 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.700310 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:50.800168 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.802879 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:50.805375 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:50.901114 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:50.923604 24233 catalog_manager.cc:4573] TS 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7bf18ffb96d04d5e9ec8ee9563812ba0: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:50.961076 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:50.961367 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:51.014921 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.102382 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:51.163393 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.163442 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.163836 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.281409 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:51.317116 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:51.322407 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:51.323385 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.334434 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:51.334582 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:51.351594 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.363440 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:51.508625 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:51.528801 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:51.534932 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.547055 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:51.698825 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.724957 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.817523 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:51.828959 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:51.846575 28260 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:36:51.846781 28260 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:36:51.847501 28260 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:36:51.849026 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:51.849489 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:51.849578 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:36:51.849851 28260 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:51.902201 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.916167 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:51.943672 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:51.976961 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:51.994220 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.003649 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:52.015740 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 21:36:52.130568 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:52.169291 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.240456 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.251964 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.296900 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:52.330521 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:52.339322 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.345803 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.357743 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.423277 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.446805 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:36:52.447760 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:52.489125 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.589916 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.804409 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.811693 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:52.820827 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:52.879233 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:52.890115 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.917588 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:52.923995 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:52.941339 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:52.957652 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.041172 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.098924 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.156430 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.214864 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:53.246973 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:53.298743 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:53.350953 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:53.443349 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:53.450712 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.473444 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.494047 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:53.532919 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.694124 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:53.718134 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.722011 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:53.755278 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.755764 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:53.784394 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.789613 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:53.888270 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:53.920389 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:53.921500 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:53.982036 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:54.087019 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.104593 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:54.144852 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:54.162743 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.174046 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.187424 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:54.226219 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:54.256716 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.312357 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:54.393427 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.486704 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:54.489919 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.501457 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:54.573658 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 21:36:54.591488 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.614924 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:54.634727 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:54.636523 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:54.656726 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:54.686493 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:54.705873 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:54.748426 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:54.748893 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:54.906111 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:54.932797 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.073026 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.080765 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.116310 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:55.175251 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:55.202543 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:55.295768 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:55.365649 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:55.413262 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:55.416325 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:36:55.417400 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.441228 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.454319 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240502 21:36:55.551623 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:55.614698 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 21:36:55.615619 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.636103 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:55.647286 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.656044 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:55.735975 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:55.794639 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.846961 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.850464 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:55.888983 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:55.935298 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.991618 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:55.995987 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:56.025009 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:56.040666 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:36:56.088965 28547 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:56.089215 28547 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:36:56.090109 28547 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:56.091920 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:56.092116 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:36:56.092206 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:36:56.092485 28547 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:56.118088 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.157790 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.179147 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.194202 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:56.222723 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:56.330121 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:56.335963 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:56.397029 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:56.524443 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:56.571033 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.629905 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:56.639815 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:56.650281 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:56.654726 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.693770 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:56.738663 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:56.746925 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.784658 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.814797 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:56.829031 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:56.832717 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:56.927983 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:57.035346 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:57.057309 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:57.075569 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:57.129719 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.130728 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:57.154278 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:57.177785 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:57.179241 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:57.224695 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.317936 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.389889 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:57.435050 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:57.454840 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 21:36:57.459811 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:36:57.460748 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:57.543872 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:57.555146 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:57.585088 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.754019 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.796032 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.880656 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:57.953337 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:57.974771 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:36:57.977030 28663 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:36:57.977209 28663 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:36:57.977913 28663 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:36:57.978991 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:57.979552 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:36:57.979756 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:57.979825 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:36:57.980057 28663 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:57.982551 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:58.022286 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.079046 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.080049 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.164332 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.226150 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:58.260615 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:58.335394 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.358755 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:58.378502 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.391757 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:58.461997 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:58.489254 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:58.515913 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:58.551870 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.649066 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.650777 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.673473 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.677155 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.701619 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:58.747087 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:58.790557 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:58.821749 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:58.853201 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:58.863984 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:58.915833 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:58.984117 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:59.092682 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:59.147379 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:59.147464 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 21:36:59.165510 28713 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:36:59.165701 28713 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:36:59.166433 28713 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:36:59.168123 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:59.168399 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:36:59.168530 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:36:59.168807 28713 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:36:59.175742 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:59.270373 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:36:59.283969 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:36:59.292523 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:59.313696 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:59.357205 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:59.378990 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:59.403487 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:36:59.424079 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:59.445014 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:36:59.488024 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:59.502470 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:59.558419 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:36:59.705189 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:36:59.714174 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:59.774925 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:36:59.806989 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:36:59.812937 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 21:36:59.847096 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:36:59.864856 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.030613 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.057873 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.065006 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.151420 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.191656 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:00.211779 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.252696 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.301471 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.355072 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:00.359593 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.366302 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.479101 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240502 21:37:00.479969 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.525748 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 21:37:00.530421 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.540490 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.599789 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.622648 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:00.649631 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:00.680174 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:37:00.769994 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:00.795807 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:00.822290 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:00.957191 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:00.969713 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.000612 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.007392 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.064529 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.076798 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.119602 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.121443 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.127306 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:01.223309 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.257685 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:01.292326 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:01.301828 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.343292 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.427927 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.429510 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.440310 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:01.566993 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:01.603386 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.710578 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:01.710702 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.730086 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:01.731720 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:01.759253 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:01.808531 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.848609 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:01.879282 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:01.962932 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.056279 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:02.067894 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.075017 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:02.146708 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.180382 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:02.216737 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:02.243682 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:02.312729 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 21:37:02.320209 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.329757 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:02.362192 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.366286 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.468561 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.522043 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.525159 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:37:02.590386 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.597178 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:02.677721 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.687528 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:37:02.712105 28816 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:02.712327 28816 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:37:02.713037 28816 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:02.714556 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.714846 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:02.714952 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:02.715246 28816 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:02.882807 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.891501 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.926385 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:02.938858 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:02.950505 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:03.009451 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:03.017385 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:37:03.024492 28833 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:37:03.024701 28833 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:03.025419 28833 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:37:03.026927 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:03.027762 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:03.027904 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:37:03.028260 28833 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:03.070739 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.103399 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:03.121134 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.156147 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.184089 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:03.262902 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:03.343344 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:03.370177 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:03.425154 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:03.503765 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:03.547935 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:03.562839 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:03.644035 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.697901 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.710762 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.787752 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.795259 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:03.826115 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.851105 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:03.895740 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:03.917022 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:03.919482 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:03.978910 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:04.013770 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.078603 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:04.092099 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:04.126364 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:04.172434 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.207865 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:04.328774 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:04.343626 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:04.443238 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:04.588035 28939 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:04.590041 28939 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:37:04.590798 28939 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:04.592823 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:04.597656 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 21:37:04.611284 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:04.613951 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:04.614200 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:04.614612 28939 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:04.618119 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:04.628630 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.675119 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:04.678382 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:04.722690 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:04.751780 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.765563 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.812073 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.822285 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:04.822996 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:04.855116 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:04.855712 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:04.858482 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:04.893985 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:04.896867 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.068921 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.075276 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:05.196815 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:05.248001 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:05.258231 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:05.396903 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.435134 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:05.484663 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 21:37:05.485759 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:05.506968 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.516906 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.583590 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.596575 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:37:05.597792 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.601531 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:05.612457 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:05.685382 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 21:37:05.686290 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.751973 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:05.755618 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:05.766170 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.939797 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:05.960882 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:06.111933 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:06.131520 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:06.135430 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.158452 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:06.169237 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:06.169936 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.189818 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.211153 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:06.303964 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:06.354804 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.366331 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:06.393343 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.443231 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:06.496312 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 21:37:06.559233 28833 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:37:06.559471 28833 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:06.560164 28833 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:37:06.561939 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:06.562063 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:06.562273 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:37:06.562703 28833 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:06.592722 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:06.605980 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.673952 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:06.674094 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:06.761564 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:06.775074 28833 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:06.775341 28833 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:06.776310 28833 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:06.777226 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:06.777796 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:06.778113 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:06.778218 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:37:06.778577 28833 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:06.913779 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:06.916740 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.005843 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.033469 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.034309 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:07.119336 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:07.157271 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 21:37:07.191983 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.226686 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.281216 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:07.313923 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.328588 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:07.345877 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:07.354876 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:07.366529 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:07.367165 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.371943 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.495632 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.509665 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.546486 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.560024 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.560849 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:37:07.561832 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.736572 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:07.746078 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:07.827016 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:07.831590 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:07.862493 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.899919 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:07.990157 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.074644 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.085181 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:08.088692 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:08.159972 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.213287 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:08.259812 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.284997 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:08.309939 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:08.452556 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.526110 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.571524 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.579008 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.580601 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.602627 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:08.620875 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:08.649533 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:08.656236 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:08.778183 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:08.790271 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.889075 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:08.900971 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:08.920943 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:08.957283 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:09.025950 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:09.026609 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:09.055845 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:09.056041 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:09.088145 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:09.114877 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:09.258078 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:09.276988 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:09.445470 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:09.527613 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:09.539492 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:09.540118 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:09.601241 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:09.635133 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 21:37:09.691047 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:09.715405 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:09.741842 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:09.778301 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:09.789896 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:09.797888 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:09.922569 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:09.922612 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:09.925412 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:09.955571 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:09.981009 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:10.029958 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.036566 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:10.169039 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:10.193363 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:10.234476 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:10.235055 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.276785 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.283659 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:10.308776 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.338692 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:10.396373 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.508828 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.545037 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:37:10.561733 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:10.590153 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:10.652282 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.658862 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:37:10.660895 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.675757 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.743921 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240502 21:37:10.744800 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.749209 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:10.770305 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:10.848991 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:10.863828 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:10.958557 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.047670 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.094262 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.137529 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:11.153489 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.209761 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.235810 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:11.297044 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.354188 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:11.364183 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.373587 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.396950 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:11.502103 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.569250 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.600178 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:11.609587 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:11.622308 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.729100 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:11.767515 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:11.806744 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:11.878577 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:11.893594 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:11.921912 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.966157 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:11.990058 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:12.008503 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:12.103971 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:12.193321 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 21:37:12.246992 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:12.250147 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.269553 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:12.300885 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:12.354514 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.365962 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:12.439659 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:12.452090 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.505782 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.547389 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:12.559736 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:12.582340 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:37:12.583549 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.588353 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:12.592042 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.612372 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:12.625587 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.692152 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.799284 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.851850 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:12.857826 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:12.875906 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:12.898504 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.916199 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:12.962967 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:12.969478 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:13.000094 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:13.237072 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.256631 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:13.292953 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.355996 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.368909 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.420464 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:13.422277 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.437494 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:13.465487 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.465920 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:13.518585 29174 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:13.518837 29174 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:37:13.519531 29174 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:13.521432 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:13.521672 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:13.521765 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:13.522043 29174 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:13.535849 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:13.741256 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.749385 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:13.782099 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.812901 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.835563 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:13.873181 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.879448 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:13.924594 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:13.960644 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:13.966936 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:14.071986 29174 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:14.072201 29174 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:14.073074 29174 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:14.074904 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:14.075049 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:14.075210 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:37:14.075470 29174 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:14.084914 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:14.090075 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:14.116416 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:14.145629 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:14.334280 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:14.335304 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:14.422508 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:14.481571 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:14.559278 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:14.571552 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:14.595595 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:14.649338 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:14.707826 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:14.793684 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:14.841779 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:14.869920 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:14.875279 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 21:37:14.892717 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:14.934741 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.000510 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.068725 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:15.074410 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:15.082502 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:15.082461 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.139721 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:15.144475 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:15.155752 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.171623 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:15.198455 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.273255 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.313566 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:15.317850 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.347302 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.411425 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:15.465044 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.480502 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:15.496101 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:15.523142 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.546649 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 172 similar messages]
W20240502 21:37:15.547696 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:15.676806 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:15.704620 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:37:15.741186 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.756618 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 21:37:15.818078 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:15.846021 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:15.929913 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.935380 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:15.943537 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.945061 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:15.983242 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:16.055773 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:16.100965 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:16.111743 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:16.209753 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:16.237074 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:16.239729 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:16.274111 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:16.296372 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:16.354018 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:16.388254 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:16.449908 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:16.481838 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:16.518761 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:16.564153 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:16.574601 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:16.622561 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:16.662635 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:16.693169 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:16.730796 29579 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:16.731022 29579 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:16.731680 29579 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:16.733282 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:16.733636 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:16.733783 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:37:16.734045 29579 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:16.780567 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:16.955885 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:16.960618 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:16.972056 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.025347 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.087936 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:17.108556 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.142685 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:17.213798 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:17.299923 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.357942 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.402805 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:17.487702 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.503963 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:17.509413 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.518147 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.528663 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.532341 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:17.597208 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 21:37:17.611861 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.632055 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 21:37:17.640375 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.654009 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:17.688477 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.690286 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:17.820998 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:17.824436 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.828431 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.887089 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:17.890141 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.902514 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:17.912653 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:17.931771 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.050678 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:18.067032 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.088604 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:18.097607 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:18.129735 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.209131 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.241122 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.287781 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:18.293099 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:18.354679 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:18.382416 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.465471 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.504199 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.561630 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:18.598397 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.599325 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.636678 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:18.763962 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:18.799966 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:18.812760 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.859390 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:18.884109 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.907301 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:18.976851 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:18.987815 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:19.084055 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.085045 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:19.137854 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:19.224092 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:19.257903 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.293277 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:19.420995 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:19.490104 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.561101 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.707159 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:19.733873 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:19.772454 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:19.774109 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:19.805395 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.835623 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:19.841882 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:19.888242 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.930063 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:19.938799 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:19.959982 24234 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 049bec57d2474c328d2e26e16a4d33a9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false replace: true } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } }
W20240502 21:37:19.987239 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:20.035506 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.060963 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.121392 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:20.141829 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 21:37:20.145328 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:20.189026 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:20.195395 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.215508 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.248102 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:20.276113 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:20.331914 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:20.375880 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.444300 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.449599 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:20.485128 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.519719 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:20.556579 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 168 similar messages]
W20240502 21:37:20.572237 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.587435 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:20.603350 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.637431 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.690512 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.721966 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:20.725904 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:37:20.730798 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:20.800594 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:37:20.826551 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:20.840901 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.000957 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:21.026711 29428 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:21.026968 29428 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:37:21.027782 29428 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:21.029318 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.029469 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:21.029578 24456 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:21.029783 29428 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:21.115653 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.140024 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.143105 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.150521 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.157768 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.158207 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.180986 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.205446 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:21.311591 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.331964 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.350062 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:21.420632 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.482756 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:21.515168 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:21.525249 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:21.542838 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:21.599746 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.658030 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.699334 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:21.729969 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:21.753703 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:21.867628 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:21.960832 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:22.021517 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.058866 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:22.179659 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:22.187003 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:22.260875 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:22.327942 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:22.339684 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.340343 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:22.403155 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:22.408766 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.438041 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:22.533279 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:22.541309 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.614722 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:22.623415 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 21:37:22.635511 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:22.644299 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:22.672304 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.689242 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:22.727306 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.821671 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 21:37:22.822603 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.827105 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.852700 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:22.921295 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:22.936635 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:22.937007 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.994289 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:22.997886 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.048578 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:23.083717 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:23.118934 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.162127 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:23.273054 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:23.380280 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:23.513648 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.546180 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:23.594188 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.629956 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 21:37:23.660290 29937 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:37:23.660501 29937 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:23.661098 29937 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:37:23.662496 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:23.662701 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:23.662811 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:37:23.663023 29937 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:23.667189 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.671429 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:23.694095 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:23.722286 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.790361 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:23.809535 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.830051 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.839990 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.893302 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:23.938812 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:23.952129 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:23.988801 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:24.008316 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:24.040746 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:24.066928 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:24.106125 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:24.181311 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:24.280560 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:24.377090 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:24.393802 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:24.409029 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:24.423723 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:24.452239 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:24.649070 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:24.734238 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:24.764971 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:24.837831 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:24.846268 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:24.886901 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:24.900615 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:24.989691 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:24.999998 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:25.033754 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:25.053411 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:25.062917 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:25.071867 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.091042 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.113893 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.149096 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:25.158810 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.203747 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.300580 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 21:37:25.319074 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:25.321269 29937 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:25.321445 29937 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:37:25.322042 29937 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:25.323644 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:25.323885 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:25.323997 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:25.324255 29937 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:25.329133 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:25.354264 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:25.392977 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.447335 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.452705 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.508335 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:25.612793 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.621851 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240502 21:37:25.645534 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:25.691287 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:25.804098 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.811123 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:25.816180 24718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 21:37:25.887619 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 21:37:25.888501 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:25.896747 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:25.942842 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:26.046114 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.047555 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:26.059628 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:26.101919 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.122339 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:26.187279 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.223455 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:26.238039 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.306560 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.318708 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.334102 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:26.397003 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:26.403270 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:26.444249 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.468176 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.570533 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:26.573048 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:26.621707 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:26.658443 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:26.748503 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:26.840205 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:26.843437 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:26.861353 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:26.905558 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:26.915236 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:27.076797 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:27.079216 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:27.154373 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:27.220963 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:27.324875 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:27.356887 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:27.387163 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:27.391078 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:27.392788 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.405421 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:27.409369 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:27.497941 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.553841 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.595651 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.622237 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:27.703789 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.717360 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:27.720113 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.765749 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:27.807653 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:27.826889 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:27.837356 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:37:27.838323 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 21:37:27.841674 30039 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:27.841874 30039 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:27.842747 30039 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:27.844475 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:27.844789 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:27.844887 24718 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:37:27.845135 30039 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:27.903115 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:27.952597 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:28.010895 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:28.019447 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.064724 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:28.066788 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 21:37:28.131731 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:28.259140 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.261066 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:28.317857 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:28.320420 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.384546 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.398990 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.498051 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:28.537915 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:28.547808 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:28.561993 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:28.606864 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:28.613889 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.643319 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.671419 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.695523 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:28.732818 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:28.803853 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.803853 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:28.856065 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:29.015964 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:29.026660 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:29.040701 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:29.055766 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:29.116719 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:29.163825 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 21:37:29.168354 30039 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:37:29.168577 30039 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:29.169252 30039 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:37:29.170877 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:29.171204 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:29.171306 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:37:29.171525 30039 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:29.177983 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:29.270939 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:29.438306 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:29.518806 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:29.518939 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:29.551757 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:29.590616 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:29.596119 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:29.613178 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:29.658771 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:29.681928 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:29.792331 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:29.826272 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:29.863339 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:29.888276 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:29.936142 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.005020 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:30.014868 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:30.080740 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:30.095083 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:30.122920 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.126564 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:30.136633 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.183851 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.252583 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.282662 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:30.283941 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.291770 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 21:37:30.372781 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:30.449312 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:30.449738 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.506403 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:30.508189 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:30.550066 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:30.636734 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:37:30.696484 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:30.772027 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.802398 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:30.844254 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:30.868688 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:30.879735 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 21:37:30.888736 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 21:37:30.892076 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.024369 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.031986 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.053515 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:31.065001 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:31.116921 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.117892 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.138821 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:31.159688 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:31.160288 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.165421 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 21:37:31.249049 30039 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:31.249382 30039 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:31.250188 30039 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:31.251858 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:31.252013 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:31.252238 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:37:31.252507 30039 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:31.257128 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:31.293236 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.296089 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:31.318954 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:31.394884 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.513736 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:31.590530 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:31.600472 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:31.655897 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:31.664947 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:31.683048 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:31.725963 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:31.795259 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:31.826820 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:32.030967 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:32.059531 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:32.060319 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:32.077689 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:32.082777 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:32.180723 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:32.184629 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:32.265270 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:32.280819 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.329883 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:32.370630 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.412710 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:32.484254 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.528928 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.589497 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:32.619143 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:32.635536 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:32.678305 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:32.721917 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.803936 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:32.818122 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.824342 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 21:37:32.843549 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.890132 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:37:32.891288 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:32.897361 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:32.920832 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:32.982323 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.057202 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:33.106678 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:33.138757 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:33.192679 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.234946 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:33.261772 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:33.270465 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.275650 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:33.357398 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.454700 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.475589 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:33.569434 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:33.569845 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.627250 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:33.636740 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.660876 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:33.719403 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.723100 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:33.757066 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.788427 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:33.828174 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.841979 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:33.956832 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:33.975644 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:33.981509 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.011011 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:34.037318 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:34.063133 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.123330 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.172137 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.292695 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:34.308571 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.332350 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.463788 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:34.571870 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.587867 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.604837 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.652887 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.817168 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.819361 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:34.828217 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:34.828933 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.852695 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:34.913616 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:34.937275 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:34.961238 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:34.978644 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:34.987848 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.007107 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.017717 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:35.032043 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:35.098850 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.103075 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.236254 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:35.246414 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:35.258128 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.268266 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:35.301631 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 21:37:35.319464 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.326090 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.362659 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:35.405670 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:35.468796 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:35.481654 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.592924 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:35.638777 24456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240502 21:37:35.654525 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:35.703593 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:35.784195 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:35.788895 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:35.880679 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:35.917754 24322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 21:37:35.937232 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 21:37:36.004021 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.004061 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.051077 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.067049 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.081383 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.091163 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.120772 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.174037 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:36.181069 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:36.181561 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.257635 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.290264 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:36.331899 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.350947 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:36.356760 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:36.368552 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.492923 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.553781 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:36.560895 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:36.573596 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.623857 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.646442 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 21:37:36.689455 30039 raft_consensus.cc:482] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:36.689654 30039 raft_consensus.cc:504] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: NON_VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: true } }
I20240502 21:37:36.690448 30039 leader_election.cc:288] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:36.692431 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
W20240502 21:37:36.692591 24718 leader_election.cc:334] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:36.692651 24718 leader_election.cc:302] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:36.692868 30039 raft_consensus.cc:2726] T ac35b080037646b4b8b34c0eab2de4e6 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:36.736619 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.788424 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.940354 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.961558 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:36.977114 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:36.994949 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:37.044988 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.077464 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.143005 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:37.175086 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.319722 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:37.319736 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:37.495548 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.496636 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.500186 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.517524 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:37.525452 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.530753 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.561057 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.580276 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:37.616328 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.665315 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:37.666723 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.673731 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:37.720166 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 21:37:37.747833 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.772011 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.797319 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:37.798259 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:37.804878 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:37.909340 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.913138 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 21:37:37.914134 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:37.953037 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:37.977303 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.025382 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 21:37:38.160090 30617 raft_consensus.cc:482] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:38.160301 30617 raft_consensus.cc:504] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:37:38.161015 30617 leader_election.cc:288] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:38.162706 24456 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:38.162858 24455 leader_election.cc:334] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:38.162954 24455 leader_election.cc:302] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f8bb40905da4891b461220b95c66832; no voters: 1769ee000eb24298a367a0de1209e6e0, 9b0952b24bd8400c9940dbd5e28417b4
I20240502 21:37:38.163187 30617 raft_consensus.cc:2726] T 566929e603d54be4bf931413f0fa1d7d P 7f8bb40905da4891b461220b95c66832 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:38.165099 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:38.179090 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:38.247669 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:38.342622 30039 raft_consensus.cc:482] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9b0952b24bd8400c9940dbd5e28417b4)
I20240502 21:37:38.342839 30039 raft_consensus.cc:504] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:38.343466 30039 leader_election.cc:288] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587), 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311)
W20240502 21:37:38.345008 24718 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:38.345149 24717 leader_election.cc:334] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:38.345294 24717 leader_election.cc:302] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 9b0952b24bd8400c9940dbd5e28417b4, b77f8f52372241d483f457389fda0f93
I20240502 21:37:38.345546 30684 raft_consensus.cc:2726] T d0ba1af1d3624d6387bf892873f3f67e P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:37:38.360702 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:38.374436 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.406296 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:38.465303 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.543504 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.546012 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:38.556711 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.645123 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:38.661384 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:38.679523 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.691606 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:38.712188 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:38.728415 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:38.804697 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:38.858088 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:38.939105 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.042227 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.042920 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.104516 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.152719 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.179203 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:39.181949 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.273574 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:39.276212 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.309770 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.421311 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.488037 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:39.518467 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.579566 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.591974 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:39.605177 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.815894 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.818534 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:39.825065 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:39.842656 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:39.882580 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:39.948746 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:39.978082 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:39.982872 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:39.988415 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 21:37:40.004457 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:40.044770 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.078655 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.097949 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.150158 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.257804 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.277068 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.305083 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:40.329094 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:40.383728 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.410327 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:40.428275 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:40.439643 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:40.500116 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:40.503708 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:40.572080 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.605310 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:40.609360 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:40.716130 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.727548 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240502 21:37:40.730621 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:40.863512 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:40.896066 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.896272 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:40.964947 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:40.964995 24717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 21:37:40.965859 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:40.999895 24323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 21:37:41.000954 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.055537 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.082422 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:41.179162 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:41.218778 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.252830 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:41.253341 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:41.258404 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.340885 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:41.364431 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:41.376005 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.400453 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:41.482969 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.527686 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.548628 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:41.552007 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.565874 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:41.739982 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:41.748332 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:41.764446 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:41.770859 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:41.805079 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:42.035457 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:42.039459 24456 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:42.098565 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.122071 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.128273 24717 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.145898 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:42.147467 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.220875 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:42.308494 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:42.344372 24717 consensus_peers.cc:485] T 2b6a4a74e0544e0e99e734a145c91d3b P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 21:37:42.365136 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.442878 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:42.463812 24586 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.490991 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.551736 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.585994 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.661723 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.723831 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.759747 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:42.770030 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.810771 24456 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.826076 24587 consensus_peers.cc:485] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:42.831645 30684 consensus_queue.cc:579] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.071s)
W20240502 21:37:42.834482 24718 consensus_peers.cc:485] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:42.911862 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
W20240502 21:37:42.958073 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:42.962628 24586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 21:37:42.963672 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:43.001044 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 21:37:43.001327 24456 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:37:43.012486 30795 consensus_queue.cc:579] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.166s)
W20240502 21:37:43.048393 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 21:37:43.096653 24455 consensus_peers.cc:485] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:37:43.105409 30861 consensus_queue.cc:579] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.250s)
I20240502 21:37:43.115768 24386 consensus_queue.cc:237] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.119040 24651 raft_consensus.cc:1261] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:37:43.119835 30920 consensus_queue.cc:1035] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:37:43.130120 30919 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
W20240502 21:37:43.130983 24323 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.131338 24651 raft_consensus.cc:2932] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.134269 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98b5668e0d504db5a7874f041c14dbfa with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 21:37:43.135119 24322 consensus_peers.cc:485] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.136945 24246 catalog_manager.cc:5484] T 98b5668e0d504db5a7874f041c14dbfa P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 12 to 13, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.156606 30919 consensus_queue.cc:579] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.033s)
I20240502 21:37:43.161085 24386 consensus_queue.cc:237] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.163710 24783 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:37:43.163812 24322 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.164269 30919 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:37:43.164965 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.167054 30920 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.167730 24783 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.171893 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5dd91818e74e42acb276a3864753094d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.173554 24246 catalog_manager.cc:5484] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.181421 29108 consensus_queue.cc:579] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.117s)
I20240502 21:37:43.231418 30919 consensus_queue.cc:579] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.071s)
I20240502 21:37:43.236172 24386 consensus_queue.cc:237] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.238391 30907 consensus_queue.cc:579] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.031s)
I20240502 21:37:43.239135 24651 raft_consensus.cc:1261] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:37:43.239790 30919 consensus_queue.cc:1035] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:37:43.242776 30920 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.243455 24651 raft_consensus.cc:2932] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
W20240502 21:37:43.244771 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:37:43.244997 24322 consensus_peers.cc:485] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adde18faca794600bb4aa24f0cf744e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.247123 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet adde18faca794600bb4aa24f0cf744e6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.249312 24246 catalog_manager.cc:5484] T adde18faca794600bb4aa24f0cf744e6 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.249996 30906 consensus_queue.cc:579] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.478s)
W20240502 21:37:43.254503 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
W20240502 21:37:43.293522 24455 consensus_peers.cc:485] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:43.336982 30795 consensus_queue.cc:579] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 836775e329f148e5908a1afc54f06428 for more than 300 seconds (300.461s)
W20240502 21:37:43.339035 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:43.340744 30795 consensus_queue.cc:579] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.492s)
W20240502 21:37:43.342293 24456 consensus_peers.cc:462] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
I20240502 21:37:43.350157 24519 consensus_queue.cc:237] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } }
I20240502 21:37:43.352623 24648 raft_consensus.cc:1261] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:37:43.353294 30918 consensus_queue.cc:1035] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240502 21:37:43.353586 24456 consensus_peers.cc:485] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.356460 30934 consensus_queue.cc:579] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.290s)
I20240502 21:37:43.356348 30795 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
I20240502 21:37:43.357194 24648 raft_consensus.cc:2932] T 9963dcea478749e0a6e076fac6537fc4 P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } }
W20240502 21:37:43.358968 24586 consensus_peers.cc:462] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:43.361677 24651 consensus_queue.cc:237] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:37:43.363433 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9963dcea478749e0a6e076fac6537fc4 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:37:43.363818 24245 catalog_manager.cc:5484] T 9963dcea478749e0a6e076fac6537fc4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 13 to 14, NON_VOTER 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:37:43.369197 24233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240502 21:37:43.369875 30795 consensus_queue.cc:579] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.492s)
W20240502 21:37:43.371906 24233 catalog_manager.cc:4573] TS 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9963dcea478749e0a6e076fac6537fc4: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111)
W20240502 21:37:43.373351 24455 consensus_peers.cc:462] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:43.374261 24386 raft_consensus.cc:1261] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:37:43.375309 30934 consensus_queue.cc:1035] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:37:43.376329 24587 consensus_peers.cc:485] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.376740 24519 consensus_queue.cc:237] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:37:43.379487 24386 raft_consensus.cc:1261] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:37:43.379796 29108 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:37:43.383008 30907 consensus_queue.cc:1035] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:37:43.384178 24456 consensus_peers.cc:485] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.384271 24386 raft_consensus.cc:2932] T 4cf38bbaaaa340f89d766a843c6bcaf9 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:37:43.385351 24234 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cf38bbaaaa340f89d766a843c6bcaf9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:37:43.386741 24245 catalog_manager.cc:5484] T 4cf38bbaaaa340f89d766a843c6bcaf9 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 7 to 8, NON_VOTER b77f8f52372241d483f457389fda0f93 (127.16.122.5) evicted. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:43.386828 30795 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:37:43.391678 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ee3718b120e4fad9ae03e47d3f1fffa with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:37:43.392189 24386 raft_consensus.cc:2932] T 2ee3718b120e4fad9ae03e47d3f1fffa P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:37:43.392637 24246 catalog_manager.cc:5484] T 2ee3718b120e4fad9ae03e47d3f1fffa P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 7 to 8, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:43.416193 30795 consensus_queue.cc:579] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.275s)
I20240502 21:37:43.420822 30684 consensus_queue.cc:579] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.084s)
I20240502 21:37:43.423954 30920 consensus_queue.cc:579] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.257s)
I20240502 21:37:43.428275 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } }
I20240502 21:37:43.430425 30907 consensus_queue.cc:579] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.182s)
I20240502 21:37:43.431011 24783 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
W20240502 21:37:43.432356 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.433069 30861 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240502 21:37:43.436180 30861 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:37:43.437026 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } }
I20240502 21:37:43.440618 24233 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:37:43.443084 24245 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 17 to 18, VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:43.448406 24386 consensus_queue.cc:237] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.451527 24783 raft_consensus.cc:1261] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
W20240502 21:37:43.451560 24322 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159): Couldn't send request to peer 7f8bb40905da4891b461220b95c66832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477d9df7d3a74b8584f3669be3256e00. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:37:43.452160 24323 consensus_peers.cc:485] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.452370 30919 consensus_queue.cc:1035] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240502 21:37:43.454638 30920 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.455188 24783 raft_consensus.cc:2932] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.459018 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 477d9df7d3a74b8584f3669be3256e00 with cas_config_opid_index 18: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.460968 24245 catalog_manager.cc:5484] T 477d9df7d3a74b8584f3669be3256e00 P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 18 to 19, NON_VOTER 7f8bb40905da4891b461220b95c66832 (127.16.122.2) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } } }
I20240502 21:37:43.498834 30918 consensus_queue.cc:579] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.095s)
W20240502 21:37:43.499254 24455 consensus_peers.cc:485] T 2e5a87719c2544a5b0cb591ed9f46df8 P 7f8bb40905da4891b461220b95c66832 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:43.516379 30905 consensus_queue.cc:579] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.251s)
I20240502 21:37:43.536075 30907 consensus_queue.cc:579] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.323s)
W20240502 21:37:43.538466 24455 consensus_peers.cc:462] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:43.540477 24519 consensus_queue.cc:237] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } }
I20240502 21:37:43.543208 24783 raft_consensus.cc:1261] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:37:43.543623 24456 consensus_peers.cc:485] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.543928 30907 consensus_queue.cc:1035] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:37:43.546907 30795 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:37:43.547564 24783 raft_consensus.cc:2932] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } }
I20240502 21:37:43.551112 24232 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c6e7eba855e48c0b0e18fa363ea01d4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:37:43.552228 24246 catalog_manager.cc:5484] T 5c6e7eba855e48c0b0e18fa363ea01d4 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 7 to 8, NON_VOTER 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8) evicted. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:43.554695 30969 ts_tablet_manager.cc:902] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:37:43.556842 30969 tablet_copy_client.cc:308] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:37:43.557868 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 5dd91818e74e42acb276a3864753094d from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:59747)
I20240502 21:37:43.558058 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 5dd91818e74e42acb276a3864753094d from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:59747: session id = 7f8bb40905da4891b461220b95c66832-5dd91818e74e42acb276a3864753094d
I20240502 21:37:43.561045 24408 tablet_copy_source_session.cc:205] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:43.562757 30969 ts_tablet_manager.cc:1875] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:37:43.568053 30969 ts_tablet_manager.cc:1888] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:37:43.568372 30969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd91818e74e42acb276a3864753094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:43.570854 30969 tablet_copy_client.cc:783] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 5 data blocks...
I20240502 21:37:43.577534 30969 tablet_copy_client.cc:647] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:43.579881 30969 tablet_copy_client.cc:523] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:43.583561 30969 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:37:43.594466 30906 consensus_queue.cc:579] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.384s)
I20240502 21:37:43.599542 30969 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:43.599994 30969 tablet_bootstrap.cc:492] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:37:43.600313 30969 ts_tablet_manager.cc:1367] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240502 21:37:43.601125 30969 raft_consensus.cc:348] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.601343 30969 raft_consensus.cc:729] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:37:43.601583 30969 consensus_queue.cc:260] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.601769 30976 ts_tablet_manager.cc:902] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:37:43.602053 30969 ts_tablet_manager.cc:1398] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:37:43.602452 30906 consensus_queue.cc:579] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.377s)
I20240502 21:37:43.602919 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-5dd91818e74e42acb276a3864753094d received from {username='slave'} at 127.16.122.2:59747
I20240502 21:37:43.603057 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 7f8bb40905da4891b461220b95c66832-5dd91818e74e42acb276a3864753094d on tablet 5dd91818e74e42acb276a3864753094d with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:37:43.606791 30905 raft_consensus.cc:482] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b77f8f52372241d483f457389fda0f93)
I20240502 21:37:43.606997 30905 raft_consensus.cc:504] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "836775e329f148e5908a1afc54f06428" member_type: VOTER last_known_addr { host: "127.16.122.6" port: 41751 } attrs { promote: false } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.607631 30905 leader_election.cc:288] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751), b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587)
W20240502 21:37:43.610520 24718 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111)
I20240502 21:37:43.611064 30976 tablet_copy_client.cc:308] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
W20240502 21:37:43.611598 24717 leader_election.cc:334] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111)
I20240502 21:37:43.611722 24717 leader_election.cc:302] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1398472b92c842f285b8c7cc43cc1527; no voters: 836775e329f148e5908a1afc54f06428, b77f8f52372241d483f457389fda0f93
I20240502 21:37:43.611851 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 98b5668e0d504db5a7874f041c14dbfa from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:59747)
I20240502 21:37:43.612025 30905 raft_consensus.cc:2726] T bdf23b7a397441b5947ce6babe3e8f62 P 1398472b92c842f285b8c7cc43cc1527 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:37:43.612025 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 98b5668e0d504db5a7874f041c14dbfa from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:59747: session id = 7f8bb40905da4891b461220b95c66832-98b5668e0d504db5a7874f041c14dbfa
I20240502 21:37:43.614629 24408 tablet_copy_source_session.cc:205] T 98b5668e0d504db5a7874f041c14dbfa P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:43.615933 30976 ts_tablet_manager.cc:1875] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:37:43.617751 24783 consensus_queue.cc:237] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
I20240502 21:37:43.620523 24651 raft_consensus.cc:1261] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer 1398472b92c842f285b8c7cc43cc1527: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20240502 21:37:43.620934 24718 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): Couldn't send request to peer e856847c3f494e6aa57f9cecf117ce4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a35079a106c4cca94cf65ffa93aefbf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.621366 30905 consensus_queue.cc:1035] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 21:37:43.621708 24717 consensus_peers.cc:485] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.621857 30907 consensus_queue.cc:579] T 8f912eb05c574368909e0b3a8edf058d P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.286s)
W20240502 21:37:43.624401 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:43.624871 30976 ts_tablet_manager.cc:1888] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 21:37:43.625325 30976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b5668e0d504db5a7874f041c14dbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:43.627822 30684 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:37:43.628855 30976 tablet_copy_client.cc:783] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 5 data blocks...
I20240502 21:37:43.628695 24651 raft_consensus.cc:2932] T 7a35079a106c4cca94cf65ffa93aefbf P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:37:43.631608 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7a35079a106c4cca94cf65ffa93aefbf with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.633975 24245 catalog_manager.cc:5484] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527 reported cstate change: config changed from index 14 to 15, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New cstate: current_term: 1 leader_uuid: "1398472b92c842f285b8c7cc43cc1527" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:43.638298 30976 tablet_copy_client.cc:647] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:43.641291 30976 tablet_copy_client.cc:523] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:43.644277 30976 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
W20240502 21:37:43.657908 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:43.661742 30976 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:43.662256 30976 tablet_bootstrap.cc:492] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:37:43.662601 30976 ts_tablet_manager.cc:1367] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.002s
I20240502 21:37:43.663414 30976 raft_consensus.cc:348] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.663676 30976 raft_consensus.cc:729] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:37:43.663956 30976 consensus_queue.cc:260] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.664580 30976 ts_tablet_manager.cc:1398] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:37:43.665437 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-98b5668e0d504db5a7874f041c14dbfa received from {username='slave'} at 127.16.122.2:59747
I20240502 21:37:43.665593 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 7f8bb40905da4891b461220b95c66832-98b5668e0d504db5a7874f041c14dbfa on tablet 98b5668e0d504db5a7874f041c14dbfa with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:37:43.691256 30918 consensus_queue.cc:579] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.400s)
I20240502 21:37:43.695647 30905 consensus_queue.cc:579] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.485s)
I20240502 21:37:43.696270 24520 consensus_queue.cc:237] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
W20240502 21:37:43.698508 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:43.699810 30861 consensus_queue.cc:579] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.439s)
W20240502 21:37:43.702152 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 21:37:43.704704 24386 consensus_queue.cc:237] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
W20240502 21:37:43.707763 24323 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 616e911db4384c17905713adc595ea0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.707800 24651 raft_consensus.cc:1261] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:37:43.708364 30861 consensus_queue.cc:1035] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:37:43.709024 24322 consensus_peers.cc:485] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.711481 30961 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:43.712282 24651 raft_consensus.cc:2932] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:43.716528 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 616e911db4384c17905713adc595ea0a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.718755 24246 catalog_manager.cc:5484] T 616e911db4384c17905713adc595ea0a P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:43.724227 24386 raft_consensus.cc:1261] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Refusing update from remote peer 7f8bb40905da4891b461220b95c66832: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 21:37:43.724435 24457 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 427345e9a65d417c84a773ac1925a18a (127.16.122.3:38427): Couldn't send request to peer 427345e9a65d417c84a773ac1925a18a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4aa176fd994cf0b55f304cf9d06fe0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.724918 30918 consensus_queue.cc:1035] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 21:37:43.725169 24455 consensus_peers.cc:485] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.727540 30795 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:37:43.730474 24232 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.730974 24386 raft_consensus.cc:2932] T 6c4aa176fd994cf0b55f304cf9d06fe0 P e856847c3f494e6aa57f9cecf117ce4d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } } }
I20240502 21:37:43.732290 24245 catalog_manager.cc:5484] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832 reported cstate change: config changed from index 9 to 10, NON_VOTER 427345e9a65d417c84a773ac1925a18a (127.16.122.3) added. New cstate: current_term: 1 leader_uuid: "7f8bb40905da4891b461220b95c66832" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:43.771574 30976 ts_tablet_manager.cc:902] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:37:43.772938 30976 tablet_copy_client.cc:308] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:37:43.773641 30986 consensus_queue.cc:579] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.550s)
I20240502 21:37:43.773681 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet adde18faca794600bb4aa24f0cf744e6 from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:59747)
I20240502 21:37:43.773859 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet adde18faca794600bb4aa24f0cf744e6 from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:59747: session id = 7f8bb40905da4891b461220b95c66832-adde18faca794600bb4aa24f0cf744e6
W20240502 21:37:43.775712 24717 consensus_peers.cc:485] T ff7e019fa82d4648afaf130d3f6343d6 P 1398472b92c842f285b8c7cc43cc1527 -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 21:37:43.776753 24408 tablet_copy_source_session.cc:205] T adde18faca794600bb4aa24f0cf744e6 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:43.777714 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d): 4840 bytes on disk
I20240502 21:37:43.777999 30920 consensus_queue.cc:579] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.521s)
I20240502 21:37:43.778074 30976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adde18faca794600bb4aa24f0cf744e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:43.778409 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(5dd91818e74e42acb276a3864753094d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 21:37:43.779721 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa): 4840 bytes on disk
I20240502 21:37:43.780301 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(98b5668e0d504db5a7874f041c14dbfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 21:37:43.783644 24386 consensus_queue.cc:237] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:37:43.784222 30976 tablet_copy_client.cc:783] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 5 data blocks...
I20240502 21:37:43.786520 24651 raft_consensus.cc:1261] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:37:43.786821 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.787236 30920 consensus_queue.cc:1035] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:37:43.787591 24323 consensus_peers.cc:485] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.789990 30961 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:43.790735 30918 consensus_queue.cc:579] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.460s)
I20240502 21:37:43.793782 24651 raft_consensus.cc:2932] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:43.794399 30976 tablet_copy_client.cc:647] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:43.796149 30976 tablet_copy_client.cc:523] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:43.798734 24246 catalog_manager.cc:5484] T 855b247f654d41c2bbb13abb25d35c1c P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 6 to 7, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:43.799225 30976 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:37:43.799726 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 855b247f654d41c2bbb13abb25d35c1c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.800482 30907 consensus_queue.cc:579] T 53c1992a21224bac883da474f143c1d7 P 7f8bb40905da4891b461220b95c66832 [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.251s)
I20240502 21:37:43.814271 30976 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:43.814640 30976 tablet_bootstrap.cc:492] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:37:43.814863 30976 ts_tablet_manager.cc:1367] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.001s
I20240502 21:37:43.815486 30976 raft_consensus.cc:348] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.815723 30976 raft_consensus.cc:729] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:37:43.815976 30976 consensus_queue.cc:260] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:43.816576 30976 ts_tablet_manager.cc:1398] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:37:43.817448 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-adde18faca794600bb4aa24f0cf744e6 received from {username='slave'} at 127.16.122.2:59747
I20240502 21:37:43.817600 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 7f8bb40905da4891b461220b95c66832-adde18faca794600bb4aa24f0cf744e6 on tablet adde18faca794600bb4aa24f0cf744e6 with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:37:43.942857 29108 consensus_queue.cc:579] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Leader has been unable to successfully communicate with peer b77f8f52372241d483f457389fda0f93 for more than 300 seconds (300.384s)
I20240502 21:37:43.947594 24651 consensus_queue.cc:237] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
W20240502 21:37:43.950680 24587 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473): Couldn't send request to peer e856847c3f494e6aa57f9cecf117ce4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82fb425a5430426f8030ba5497736b11. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.951366 24520 raft_consensus.cc:1261] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Refusing update from remote peer 427345e9a65d417c84a773ac1925a18a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240502 21:37:43.951874 24586 consensus_peers.cc:485] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a -> Peer b77f8f52372241d483f457389fda0f93 (127.16.122.5:42587): Couldn't send request to peer b77f8f52372241d483f457389fda0f93. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.5:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:43.951905 31022 consensus_queue.cc:1035] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:37:43.954870 29108 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:37:43.955677 24520 raft_consensus.cc:2932] T 82fb425a5430426f8030ba5497736b11 P 7f8bb40905da4891b461220b95c66832 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } } }
I20240502 21:37:43.958088 24234 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 82fb425a5430426f8030ba5497736b11 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:43.960341 24248 catalog_manager.cc:5484] T 82fb425a5430426f8030ba5497736b11 P 427345e9a65d417c84a773ac1925a18a reported cstate change: config changed from index 12 to 13, NON_VOTER e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1) added. New cstate: current_term: 1 leader_uuid: "427345e9a65d417c84a773ac1925a18a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b77f8f52372241d483f457389fda0f93" member_type: VOTER last_known_addr { host: "127.16.122.5" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:37:43.964233 24456 consensus_peers.cc:485] T a8a9b14a7ff04ddab0d1e1a8e9fcb9b0 P 7f8bb40905da4891b461220b95c66832 -> Peer 836775e329f148e5908a1afc54f06428 (127.16.122.6:41751): Couldn't send request to peer 836775e329f148e5908a1afc54f06428. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.6:41751: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:43.984298 30976 ts_tablet_manager.cc:902] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:37:43.985723 30976 tablet_copy_client.cc:308] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:37:43.986526 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 477d9df7d3a74b8584f3669be3256e00 from peer 7f8bb40905da4891b461220b95c66832 ({username='slave'} at 127.16.122.2:59747)
I20240502 21:37:43.986711 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 477d9df7d3a74b8584f3669be3256e00 from peer 7f8bb40905da4891b461220b95c66832 at {username='slave'} at 127.16.122.2:59747: session id = 7f8bb40905da4891b461220b95c66832-477d9df7d3a74b8584f3669be3256e00
I20240502 21:37:43.989570 24408 tablet_copy_source_session.cc:205] T 477d9df7d3a74b8584f3669be3256e00 P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:43.991400 24520 raft_consensus.cc:1203] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:37:43.991681 30976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477d9df7d3a74b8584f3669be3256e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:44.000370 30976 tablet_copy_client.cc:783] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 5 data blocks...
W20240502 21:37:44.004652 24717 consensus_peers.cc:485] T e847a796029d4edea79694f5402a291a P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:44.008396 30976 tablet_copy_client.cc:647] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:44.010324 30976 tablet_copy_client.cc:523] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:44.013074 30976 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: Bootstrap starting.
I20240502 21:37:44.033330 30976 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:44.033756 30976 tablet_bootstrap.cc:492] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: Bootstrap complete.
I20240502 21:37:44.033852 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6): 4586 bytes on disk
I20240502 21:37:44.034034 30976 ts_tablet_manager.cc:1367] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.003s
I20240502 21:37:44.034536 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(adde18faca794600bb4aa24f0cf744e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 21:37:44.034868 30976 raft_consensus.cc:348] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:44.035059 30976 raft_consensus.cc:729] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f8bb40905da4891b461220b95c66832, State: Initialized, Role: LEARNER
I20240502 21:37:44.035254 30976 consensus_queue.cc:260] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: NON_VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: true } }
I20240502 21:37:44.035647 30976 ts_tablet_manager.cc:1398] T 477d9df7d3a74b8584f3669be3256e00 P 7f8bb40905da4891b461220b95c66832: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:37:44.036442 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 7f8bb40905da4891b461220b95c66832-477d9df7d3a74b8584f3669be3256e00 received from {username='slave'} at 127.16.122.2:59747
I20240502 21:37:44.036612 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 7f8bb40905da4891b461220b95c66832-477d9df7d3a74b8584f3669be3256e00 on tablet 477d9df7d3a74b8584f3669be3256e00 with peer 7f8bb40905da4891b461220b95c66832
I20240502 21:37:44.044034 31031 ts_tablet_manager.cc:902] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: Initiating tablet copy from peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687)
I20240502 21:37:44.046170 31031 tablet_copy_client.cc:308] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.4:33687
I20240502 21:37:44.046974 24804 tablet_copy_service.cc:140] P 1398472b92c842f285b8c7cc43cc1527: Received BeginTabletCopySession request for tablet 7a35079a106c4cca94cf65ffa93aefbf from peer e856847c3f494e6aa57f9cecf117ce4d ({username='slave'} at 127.16.122.1:35481)
I20240502 21:37:44.047127 24804 tablet_copy_service.cc:161] P 1398472b92c842f285b8c7cc43cc1527: Beginning new tablet copy session on tablet 7a35079a106c4cca94cf65ffa93aefbf from peer e856847c3f494e6aa57f9cecf117ce4d at {username='slave'} at 127.16.122.1:35481: session id = e856847c3f494e6aa57f9cecf117ce4d-7a35079a106c4cca94cf65ffa93aefbf
I20240502 21:37:44.049903 24804 tablet_copy_source_session.cc:205] T 7a35079a106c4cca94cf65ffa93aefbf P 1398472b92c842f285b8c7cc43cc1527: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:44.051700 31031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a35079a106c4cca94cf65ffa93aefbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:44.053969 24520 raft_consensus.cc:1203] T 98b5668e0d504db5a7874f041c14dbfa P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240502 21:37:44.059219 31031 tablet_copy_client.cc:783] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Starting download of 5 data blocks...
I20240502 21:37:44.067332 31031 tablet_copy_client.cc:647] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:44.069638 31031 tablet_copy_client.cc:523] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:44.072744 31031 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap starting.
W20240502 21:37:44.079767 24455 consensus_peers.cc:485] T dc5c40e2bf0340958ecd3e7541209e95 P 7f8bb40905da4891b461220b95c66832 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 21:37:44.092913 31037 ts_tablet_manager.cc:902] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: Initiating tablet copy from peer e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473)
I20240502 21:37:44.094447 31037 tablet_copy_client.cc:308] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.1:36473
I20240502 21:37:44.095245 24408 tablet_copy_service.cc:140] P e856847c3f494e6aa57f9cecf117ce4d: Received BeginTabletCopySession request for tablet 616e911db4384c17905713adc595ea0a from peer 1398472b92c842f285b8c7cc43cc1527 ({username='slave'} at 127.16.122.4:44345)
I20240502 21:37:44.095435 24408 tablet_copy_service.cc:161] P e856847c3f494e6aa57f9cecf117ce4d: Beginning new tablet copy session on tablet 616e911db4384c17905713adc595ea0a from peer 1398472b92c842f285b8c7cc43cc1527 at {username='slave'} at 127.16.122.4:44345: session id = 1398472b92c842f285b8c7cc43cc1527-616e911db4384c17905713adc595ea0a
I20240502 21:37:44.096633 31031 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:44.097039 31031 tablet_bootstrap.cc:492] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: Bootstrap complete.
I20240502 21:37:44.097402 31031 ts_tablet_manager.cc:1367] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240502 21:37:44.097985 24408 tablet_copy_source_session.cc:205] T 616e911db4384c17905713adc595ea0a P e856847c3f494e6aa57f9cecf117ce4d: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:44.098129 31031 raft_consensus.cc:348] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
I20240502 21:37:44.098459 31031 raft_consensus.cc:729] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e856847c3f494e6aa57f9cecf117ce4d, State: Initialized, Role: LEARNER
I20240502 21:37:44.098740 31031 consensus_queue.cc:260] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: NON_VOTER last_known_addr { host: "127.16.122.1" port: 36473 } attrs { promote: true } }
I20240502 21:37:44.099376 31031 ts_tablet_manager.cc:1398] T 7a35079a106c4cca94cf65ffa93aefbf P e856847c3f494e6aa57f9cecf117ce4d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:37:44.099524 31037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e911db4384c17905713adc595ea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:44.100389 24804 tablet_copy_service.cc:342] P 1398472b92c842f285b8c7cc43cc1527: Request end of tablet copy session e856847c3f494e6aa57f9cecf117ce4d-7a35079a106c4cca94cf65ffa93aefbf received from {username='slave'} at 127.16.122.1:35481
I20240502 21:37:44.100502 24804 tablet_copy_service.cc:433] P 1398472b92c842f285b8c7cc43cc1527: ending tablet copy session e856847c3f494e6aa57f9cecf117ce4d-7a35079a106c4cca94cf65ffa93aefbf on tablet 7a35079a106c4cca94cf65ffa93aefbf with peer e856847c3f494e6aa57f9cecf117ce4d
I20240502 21:37:44.105917 31037 tablet_copy_client.cc:783] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 5 data blocks...
I20240502 21:37:44.112556 31037 tablet_copy_client.cc:647] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:44.114807 31037 tablet_copy_client.cc:523] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:44.118142 31037 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: Bootstrap starting.
I20240502 21:37:44.138882 31037 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:44.139376 31037 tablet_bootstrap.cc:492] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: Bootstrap complete.
I20240502 21:37:44.139706 31037 ts_tablet_manager.cc:1367] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.005s
I20240502 21:37:44.141036 31037 raft_consensus.cc:348] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:37:44.141382 31037 raft_consensus.cc:729] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1398472b92c842f285b8c7cc43cc1527, State: Initialized, Role: LEARNER
I20240502 21:37:44.141652 31037 consensus_queue.cc:260] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } } peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:37:44.142247 31037 ts_tablet_manager.cc:1398] T 616e911db4384c17905713adc595ea0a P 1398472b92c842f285b8c7cc43cc1527: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:37:44.143206 24408 tablet_copy_service.cc:342] P e856847c3f494e6aa57f9cecf117ce4d: Request end of tablet copy session 1398472b92c842f285b8c7cc43cc1527-616e911db4384c17905713adc595ea0a received from {username='slave'} at 127.16.122.4:44345
I20240502 21:37:44.143347 24408 tablet_copy_service.cc:433] P e856847c3f494e6aa57f9cecf117ce4d: ending tablet copy session 1398472b92c842f285b8c7cc43cc1527-616e911db4384c17905713adc595ea0a on tablet 616e911db4384c17905713adc595ea0a with peer 1398472b92c842f285b8c7cc43cc1527
I20240502 21:37:44.144297 30961 consensus_queue.cc:579] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.231s)
I20240502 21:37:44.148627 30861 consensus_queue.cc:579] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Leader has been unable to successfully communicate with peer 1769ee000eb24298a367a0de1209e6e0 for more than 300 seconds (300.235s)
I20240502 21:37:44.151408 24384 consensus_queue.cc:237] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
W20240502 21:37:44.155604 24322 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Couldn't send request to peer 1769ee000eb24298a367a0de1209e6e0. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:44.155800 24651 raft_consensus.cc:1261] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 21:37:44.156456 31051 consensus_queue.cc:1035] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240502 21:37:44.158839 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4:33687): Couldn't send request to peer 1398472b92c842f285b8c7cc43cc1527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d8a25580bc74004b0e31c34f841d2cc. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:37:44.158838 24233 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader e856847c3f494e6aa57f9cecf117ce4d (127.16.122.1:36473) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } }.
Pending config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } }
I20240502 21:37:44.160497 24437 maintenance_manager.cc:419] P e856847c3f494e6aa57f9cecf117ce4d: Scheduling UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf): 4925 bytes on disk
W20240502 21:37:44.161218 24323 consensus_peers.cc:485] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:44.161126 24326 maintenance_manager.cc:643] P e856847c3f494e6aa57f9cecf117ce4d: UndoDeltaBlockGCOp(7a35079a106c4cca94cf65ffa93aefbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 21:37:44.162313 30920 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:44.166260 24233 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 16: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:37:44.166147 24651 raft_consensus.cc:2932] T 9d8a25580bc74004b0e31c34f841d2cc P 427345e9a65d417c84a773ac1925a18a [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } } }
I20240502 21:37:44.168704 24246 catalog_manager.cc:5484] T 9d8a25580bc74004b0e31c34f841d2cc P e856847c3f494e6aa57f9cecf117ce4d reported cstate change: config changed from index 16 to 17, NON_VOTER 1398472b92c842f285b8c7cc43cc1527 (127.16.122.4) added. New cstate: current_term: 1 leader_uuid: "e856847c3f494e6aa57f9cecf117ce4d" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: NON_VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: NON_VOTER last_known_addr { host: "127.16.122.4" port: 33687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:37:44.192778 24233 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 9d8a25580bc74004b0e31c34f841d2cc with cas_config_opid_index 16: aborting the task: latest config opid_index 17; task opid_index 16
W20240502 21:37:44.247936 24717 consensus_peers.cc:462] T 7b2661e0b5fe4a0b94c150b14b8fd9ec P 1398472b92c842f285b8c7cc43cc1527 -> Peer 1769ee000eb24298a367a0de1209e6e0 (127.16.122.8:37045): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.122.8:37045: connect: Connection refused (error 111)
I20240502 21:37:44.252349 31057 ts_tablet_manager.cc:902] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: Initiating tablet copy from peer 7f8bb40905da4891b461220b95c66832 (127.16.122.2:43159)
I20240502 21:37:44.255379 31057 tablet_copy_client.cc:308] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.2:43159
I20240502 21:37:44.256119 24541 tablet_copy_service.cc:140] P 7f8bb40905da4891b461220b95c66832: Received BeginTabletCopySession request for tablet 6c4aa176fd994cf0b55f304cf9d06fe0 from peer 427345e9a65d417c84a773ac1925a18a ({username='slave'} at 127.16.122.3:58123)
I20240502 21:37:44.256260 24541 tablet_copy_service.cc:161] P 7f8bb40905da4891b461220b95c66832: Beginning new tablet copy session on tablet 6c4aa176fd994cf0b55f304cf9d06fe0 from peer 427345e9a65d417c84a773ac1925a18a at {username='slave'} at 127.16.122.3:58123: session id = 427345e9a65d417c84a773ac1925a18a-6c4aa176fd994cf0b55f304cf9d06fe0
I20240502 21:37:44.257241 31022 consensus_queue.cc:579] T 049bec57d2474c328d2e26e16a4d33a9 P 427345e9a65d417c84a773ac1925a18a [LEADER]: Leader has been unable to successfully communicate with peer 9b0952b24bd8400c9940dbd5e28417b4 for more than 300 seconds (300.294s)
I20240502 21:37:44.259789 24541 tablet_copy_source_session.cc:205] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 7f8bb40905da4891b461220b95c66832: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 21:37:44.261571 31057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4aa176fd994cf0b55f304cf9d06fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:37:44.267264 31057 tablet_copy_client.cc:783] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 5 data blocks...
I20240502 21:37:44.275348 31057 tablet_copy_client.cc:647] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:37:44.277356 31057 tablet_copy_client.cc:523] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:37:44.281823 31057 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: Bootstrap starting.
I20240502 21:37:44.286964 24569 maintenance_manager.cc:419] P 7f8bb40905da4891b461220b95c66832: Scheduling UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00): 4586 bytes on disk
I20240502 21:37:44.287583 24459 maintenance_manager.cc:643] P 7f8bb40905da4891b461220b95c66832: UndoDeltaBlockGCOp(477d9df7d3a74b8584f3669be3256e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 21:37:44.300925 31057 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:37:44.301347 31057 tablet_bootstrap.cc:492] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: Bootstrap complete.
I20240502 21:37:44.301647 31057 ts_tablet_manager.cc:1367] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 21:37:44.302299 31057 raft_consensus.cc:348] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:37:44.302558 31057 raft_consensus.cc:729] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 427345e9a65d417c84a773ac1925a18a, State: Initialized, Role: LEARNER
I20240502 21:37:44.302785 31057 consensus_queue.cc:260] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } } peers { permanent_uuid: "1769ee000eb24298a367a0de1209e6e0" member_type: VOTER last_known_addr { host: "127.16.122.8" port: 37045 } attrs { promote: false } } peers { permanent_uuid: "427345e9a65d417c84a773ac1925a18a" member_type: NON_VOTER last_known_addr { host: "127.16.122.3" port: 38427 } attrs { promote: true } }
I20240502 21:37:44.303324 31057 ts_tablet_manager.cc:1398] T 6c4aa176fd994cf0b55f304cf9d06fe0 P 427345e9a65d417c84a773ac1925a18a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:37:44.304194 24541 tablet_copy_service.cc:342] P 7f8bb40905da4891b461220b95c66832: Request end of tablet copy session 427345e9a65d417c84a773ac1925a18a-6c4aa176fd994cf0b55f304cf9d06fe0 received from {username='slave'} at 127.16.122.3:58123
I20240502 21:37:44.305780 24541 tablet_copy_service.cc:433] P 7f8bb40905da4891b461220b95c66832: ending tablet copy session 427345e9a65d417c84a773ac1925a18a-6c4aa176fd994cf0b55f304cf9d06fe0 on tablet 6c4aa176fd994cf0b55f304cf9d06fe0 with peer 427345e9a65d417c84a773ac1925a18a
I20240502 21:37:44.323215 24520 raft_consensus.cc:1203] T adde18faca794600bb4aa24f0cf744e6 P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:37:44.333315 24699 maintenance_manager.cc:419] P 427345e9a65d417c84a773ac1925a18a: Scheduling UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0): 4925 bytes on disk
I20240502 21:37:44.333925 24830 maintenance_manager.cc:419] P 1398472b92c842f285b8c7cc43cc1527: Scheduling UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a): 4841 bytes on disk
I20240502 21:37:44.334074 24590 maintenance_manager.cc:643] P 427345e9a65d417c84a773ac1925a18a: UndoDeltaBlockGCOp(6c4aa176fd994cf0b55f304cf9d06fe0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 21:37:44.334652 24721 maintenance_manager.cc:643] P 1398472b92c842f285b8c7cc43cc1527: UndoDeltaBlockGCOp(616e911db4384c17905713adc595ea0a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240502 21:37:44.384681 30861 raft_consensus.cc:1050] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d: attempting to promote NON_VOTER 7f8bb40905da4891b461220b95c66832 to VOTER
I20240502 21:37:44.385542 30861 consensus_queue.cc:237] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } }
I20240502 21:37:44.387864 24520 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 7f8bb40905da4891b461220b95c66832 [term 1 LEARNER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:37:44.388599 24783 raft_consensus.cc:1261] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Refusing update from remote peer e856847c3f494e6aa57f9cecf117ce4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:37:44.389444 30920 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:37:44.389868 24323 consensus_peers.cc:485] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d -> Peer 9b0952b24bd8400c9940dbd5e28417b4 (127.16.122.7:36311): Couldn't send request to peer 9b0952b24bd8400c9940dbd5e28417b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.7:36311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:37:44.389982 30920 consensus_queue.cc:1035] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:37:44.396379 30920 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P e856847c3f494e6aa57f9cecf117ce4d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2" port: 43159 } attrs { promote: false } } }
I20240502 21:37:44.397186 24783 raft_consensus.cc:2932] T 5dd91818e74e42acb276a3864753094d P 1398472b92c842f285b8c7cc43cc1527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f8bb40905da4891b461220b95c66832 (127.16.122.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e856847c3f494e6aa57f9cecf117ce4d" member_type: VOTER last_known_addr { host: "127.16.122.1" port: 36473 } } peers { permanent_uuid: "1398472b92c842f285b8c7cc43cc1527" member_type: VOTER last_known_addr { host: "127.16.122.4" port: 33687 } } peers { permanent_uuid: "9b0952b24bd8400c9940dbd5e28417b4" member_type: VOTER last_known_addr { host: "127.16.122.7" port: 36311 } attrs { promote: false } } peers { permanent_uuid: "7f8bb40905da4891b461220b95c66832" member_type: VOTER last_known_addr { host: "127.16.122.2"